Last week I attended an interesting session on Exchange 2010 monitoring and backup with System Center host by the System Center Influencer guys (https://connect.microsoft.com/SystemCenterCommunity?wa=wsignin1.0).

Two main topics has been covered:

1. The new Exchange 2010 Management Pack for OpsMgr
2. Using DPM to backup Exchange 2010

In this post I will focus on the first topic, but I want to add that you should take a serious look at DPM and when you do this, please try to have a new perspective. Do not try to map what you’re currently doing with your backup and recovery solution to DPM, rather you should learn DPM paradigm and see your data assets in this new perspective. It is a new backup paradigm and you’ll need to do things differently… the worst DPM implementations I saw were the ones where DPM has been used like a "traditional" backup software. Not the way to go.

The only pitfalls I must make you aware of is that you need to wait for the next version of DPM to protect Exchange 2010, and this is a pity given the tons of functionality in DPM and the fact that current DPM users must delay Exchange 2010 adoption ’til the release of DPM.

Returning to OpsMgr, the MP has been released soon after Exchange 2010 delivery, in this case we have no road block. We can adopt Exchange 2010 and being able to monitor it with our current OpsMgr infrastructure.

SO what’s new in this MP:

· Alert correlation

· Alert classification

· Mail flow statistics reporting

· Full protocol synthetic transaction coverage

· Service-oriented reporting

· Exchange aware availability modeling

The most interesting stuff in there is the new correlation engine, it’s a stand alone service that help reducing noise and who is in charge of generating alerts into OpsMgr via the SDK. A brand new approach in MPs and something I will return on in future posts.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

Our outsourced service based on OpsMgr uses a gateway infrastructure to monitor customers sites. Every customer has an appropriate number of gateways reporting to our data center. We used the site concept to map our customers, so every customer is a site. When we approve the gateway we specify the site name. From time to time gateways need to be replaced, moved or added.

Replacing a gateway with a new one is the main topic of this article, and no, it’s not easy nor straightforward.

Let’s recap the scenario:

• gateway server needs to be replaced
• agents are assigned to the gateway at install time, no Active Directory integration
• gateway is associated to a Site at approval time

The common action plan is the following:

1. add a new gateway ad associate it to the same Site of the old one
2. from OpsMgr console move the agents to the newly added gateway
3. uninstall the old gateway
4. remove the gateway approval with microsoft.enterprisemanagement.gatewayapprovaltool and action=delete

Adding a new gateway is a no brainer, the product documentation is clear enough so I won’t spend time on it. Moving the agent to a new MS is straightforward as well, and you can do it from the UI, right? Wrong.

Let’s start with a quick background on how Management Servers (MS) and agents work together, remember the gateway is a type of MS. When you change the MS agent for an agent, the old one removes the agent from the managed agents, the new one, on the other side, adds the agent to the authorized list. If the agent is not in the authorized list the MS won’t respond to it and it will register an event in the event log saying the agent is not authorized for this MS. Agents get their configuration information from the assigned MS a MS switch is part of this configuration. Now the question is: how can an agent learn it should refer to a new MS? In this situation simply it cannot. We have closed our agent out of the door. The agent cannot receive any configuration from old MS because it is not in the authorized list, so it won’t learn it should get its configuration from the new MS. I saw a few agents trying to refer to the RMS in this situation, saying something like "failing over to secondary MS (RMS)", I don’t know if this is the expected behavior to avoid this deadlock, but for sure it cannot work for gateways. Gateways are in untrusted domains their managed agents are not able to reach the RMS.

If you are in this situation, the first thing to do is to regain control of your agents, to do so you should reset your agents to the old gateway.

Now that we’re in control once again let’s do a different action plan:

1. add the new MS to the secondary MS of the agents
2. Wait for the new configuration to propagate (21025 event)
3. Switch the primary and secondary server so that the new MS will become the primary one.
4. Remove the old MS from the agents

This can be done via Active Directory integration or via powershell. Since we cannot use AD, let’s share a simple powershell script:

#old ms
$msp=Get-ManagementServer | where {$_.Name -eq ‘gw1.somedomain.it’}
#new ms
$ms=Get-ManagementServer | where {$_.Name -eq ‘gw2.somedomain.it}
$failoverServers = New-Object System.Collections.Generic.List“1"[[Microsoft.EnterpriseManagement.Administration.ManagementServer,Microsoft.EnterpriseManagement.OperationsManager,Version=6.0.4900.0,Culture=neutral,PublicKeyToken=31bf3856ad364e35]]"
$failoverServers.Add($ms)
$agents = $msp.GetAgentManagedComputers()
foreach ($a in $agents)
{
    $a.SetManagementServers($msp, $failoverServers)
}

The script add the new gateway ($ms) as a failover server to the agents managed by the old one ($msp). You can then test the move from UI. Once you have moved a few agents and checked their working as expected with the new gateway, you can move over the other agents. But first remember to give enough time to the agents to get the new configuration (they must learn the new failover MS list), if not you will close the agent out of the door once again.

# wait for 21025 on every MS and agent and reset the primary GW and the failover MS list
$agents = $msp.GetAgentManagedComputers()

foreach ($a in $agents)
{
    $a.SetManagementServers($ms, $null)
}

Steps 1 and 2 completed.

Removing the gateway binaries from the old gateway is a matter of running the uninstall procedure, by the way you can simply turn off the gateway and you’ll get the same effect. In either case you must manually remove the gateway from the approved gateways list of your OpsMgr management group.

Let’s move on to step 4. Using the gateway approval tool with /Action=delete we are supposed to be able to remove the gateway:

Interesting, isn’t it? We just removed all the agents from the gateway management space, but still we got this error, and the gateway stays there.

Using SQL trace is simple to track down the steps performed by the tool:

1. it looks for a Microsoft.SystemCenter.GatewayManagementServer with the given gateway name
2. it looks for relationships of type Microsoft.SystemCenter.HealthServiceCommunication and Microsoft.SystemCenter.HealthServiceShouldManageEntity related to the gateway

Replaying the queries becomes evident the gateway has still a relationship of type HealthServiceShouldManageEntity with the associated Site. To remove the gateway we must remove that relationship, but we have no supported way to do that, at least as far as I know. So this is my *unsupported* way to decommission a gateway when it has been associated with a Site.

First check if we really are in the situation where the only relationship still in place is the one related to the site (if not we must check what’s left). For this reason I will split the query in two, the first part will return a list of relationships and the second one will mark the relationship deleted:

declare @nodeHS nvarchar(255)

Set @nodeHS=N’Microsoft.SystemCenter.HealthService:gateway.somedomain.it’

(SELECT *

FROM dbo.RelationshipGenericView

WHERE ((RelationshipGenericView.[MonitoringRelationshipClassId] = dbo.fn_ManagedTypeId_MicrosoftSystemCenterHealthServiceShouldManageEntity()) AND (((dbo.[RelationshipGenericView].[IsDeleted] = 0))))

AND ([RelationshipGenericView].[SourceMonitoringObjectId]

IN (select BaseManagedEntityId from BaseManagedEntity where FullName =@NodeHS)))

If we have just one releationship left with the site, we can mark the relationship deleted (unsupported, you know):

update Relationship set IsDeleted=1 where RelationshipId=’Relationship GUID returned by the previous query’

In conclusion, decommissioning and replacing a gateway server has some caveats, if you add to the equation a site relationship then you need to perform the above hack.

Let me know if this works for you.

For future reference I list the queries performed by the gateway approval tool:

exec sp_executesql N’– MTV_SelectProperty_c1721bcc-35f7-5a49-5d5f-6880687c3d48 <ManagedTypeId,PrincipalName0>

SELECT [MTV_HealthService].[BaseManagedEntityId], [MTV_HealthService].[DisplayName_55270A70_AC47_C853_C617_236B0CFF9B4C], [MTV_HealthService].[ActionAccountIdentity], [MTV_HealthService].[ActiveDirectoryManaged], [MTV_HealthService].[AuthenticationName], [MTV_HealthService].[CreateListener], [MTV_HealthService].[HeartbeatEnabled], [MTV_HealthService].[HeartbeatInterval], [MTV_HealthService].[InstalledBy], [MTV_HealthService].[InstallTime], [MTV_HealthService].[IsAgent], [MTV_HealthService].[IsGateway], [MTV_HealthService].[IsManagementServer], [MTV_HealthService].[IsManuallyInstalled], [MTV_HealthService].[IsRHS], [MTV_HealthService].[MaximumQueueSize], [MTV_HealthService].[MaximumSizeOfAllTransferredFiles], [MTV_HealthService].[PatchList], [MTV_HealthService].[Port], [MTV_HealthService].[ProxyingEnabled], [MTV_HealthService].[RequestCompression], [MTV_HealthService].[Version], [MTV_HealthService].[AutoApproveManuallyInstalledAgents_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[ManagementServerSCP_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[NumberOfMissingHeartBeatsToMarkMachineDown_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[ProxyAddress_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[ProxyPort_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[RejectManuallyInstalledAgents_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[UseProxyServer_9189A49E_B2DE_CAB0_2E4F_4925B68E335D], [MTV_HealthService].[WebConsoleUrl_F9069CA9_A790_E274_0C2C_DE210E57F67C], [MTV_HealthService].[SiteId_CECAAFDA_33B6_B628_0CDA_445E21B7291D], [MTV_HealthService].[SiteName_CECAAFDA_33B6_B628_0CDA_445E21B7291D], [MTV_HealthService].[PrincipalName] FROM dbo.[MTV_HealthService]

INNER JOIN dbo.[TypedManagedEntity] AS TME ON TME.[BaseManagedEntityId] = [MTV_HealthService].[BaseManagedEntityId]

WHERE (MTV_HealthService.[PrincipalName] LIKE @PrincipalName0) AND (((TME.[ManagedTypeId] = @ManagedTypeId)))’,N’@ManagedTypeId uniqueidentifier,@PrincipalName0 ntext’,@ManagedTypeId=’C1721BCC-35F7-5A49-5D5F-6880687C3D48′,@PrincipalName0=N’gateways.somedomain.it’

– TypeID ‘C1721BCC-35F7-5A49-5D5F-6880687C3D48′ = Microsoft.SystemCenter.GatewayManagementServer

exec sp_executesql N’– RelationshipWithCriteria <TargetEntityId0,IsDeleted0,RelationshipTypeId0>

SELECT [Relationship].[RelationshipId], [Relationship].[TargetEntityId], [Relationship].[IsDeleted] FROM dbo.Relationship

WHERE Relationship.[TargetEntityId] = @TargetEntityId0 AND Relationship.[IsDeleted] = @IsDeleted0 AND Relationship.[RelationshipTypeId] = @RelationshipTypeId0′,N’@TargetEntityId0 uniqueidentifier,@IsDeleted0 bit,@RelationshipTypeId0 uniqueidentifier’,@TargetEntityId0=’FC75E426-26C5-B237-FE9F-F14F540CCB0E’,@IsDeleted0=0,@RelationshipTypeId0=’37848E16-37A2-B81B-DAAF-60A5A626BE93′

– Relationship Microsoft.SystemCenter.HealthServiceCommunication

exec sp_executesql N’– RelationshipWithCriteria <SourceEntityId0,IsDeleted0,RelationshipTypeId0>

SELECT [Relationship].[RelationshipId], [Relationship].[SourceEntityId], [Relationship].[IsDeleted] FROM dbo.Relationship

WHERE Relationship.[SourceEntityId] = @SourceEntityId0 AND Relationship.[IsDeleted] = @IsDeleted0 AND Relationship.[RelationshipTypeId] = @RelationshipTypeId0′,N’@SourceEntityId0 uniqueidentifier,@IsDeleted0 bit,@RelationshipTypeId0 uniqueidentifier’,@SourceEntityId0=’FC75E426-26C5-B237-FE9F-F14F540CCB0E’,@IsDeleted0=0,@RelationshipTypeId0=’2F71C644-E092-B80A-040B-5C81BA1EC353′

– Relationship Microsoft.SystemCenter.HealthServiceShouldManageEntity

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

A few week ago a customer of ours has been hit by a time issue, the internal time reference server jumped to anno domini 2020. Since this happened during the weekend it took a few hours to be fixed, in the meantime opsmgr agents did their job posting data to the opsmgr infrastructure. The net result was a bunch of monitors unhealthy with last changed time set to 2020. Annoying I thought I need to reset all the broken monitors via powershell. Alas this was not only annoying, but blocking as well. Monitors won’t reset nor their state change in any case.

First consideration the opsmgr data access layer should block any data insertion with a too large time skew from its own date time reference (10’ to 15’ should be the maximum threshold immo).

Anyway, time for some reverse engineering once again.

First of all a few queries to identify the bogus state data. We have two tables involved here StateChangeEvent and State. The former collects all event state events, the ones you can check in health explorer, the latter reports the last known state for any given managed entity / monitor pair.

Easy enough, let check for all data updated after December 31st

select * from dbo.StateChangeEvent
where TimeGenerated > ‘12-31-2009′

select ME.FullName, M.MonitorName, State.* from dbo.State with (nolock)
inner join dbo.BaseManagedEntity ME with (nolock) on ME.BaseManagedEntityId=State.BaseManagedEntityId
inner join dbo.Monitor M with (nolock) on M.MonitorId=State.MonitorId
where State.LastModified > ‘12-31-2009′

Obviously my first though has been lets modify the LastModified field, but here we’re in the unsupported realm and before any mods a further analisys of the insight working needs to be accomplished. The core stoerd procedure for any stage change turned to be 

PROCEDURE [dbo].[p_StateChangeEventProcess]
(      
    @BaseManagedEntityId uniqueidentifier,
    @EventOriginId uniqueidentifier,
    @MonitorId uniqueidentifier,
    @NewHealthState tinyint,
    @OldHealthState tinyint,    
    @TimeGenerated datetime,
    @Context nvarchar(max) = NULL
)

this one in turns calls

PROCEDURE [dbo].[p_StateUpsert]
(
    @BaseManagedEntityId uniqueidentifier,
    @MonitorId uniqueidentifier,
    @HealthState tinyint,
    @LastModified datetime
)

and if p_StateUpsert returns with success it will insert a row in the StateChangeEvent table.

p_StateUpsert, among other checks, sets a control on the state update date time if it is earlier in the timeline respect the last time a monitor state has been updated the state change is discarded. This makes sense since state change are not guaranteed to arrive in chronological order. At the same time without a control on a time skew we can have a dos here.

Anyway from my analysis the LastModified field can be safely changed (still unsupported realm):

update dbo.StateChangeEvent set TimeGenerated=TimeAdded
where TimeGenerated > ‘12-31-2009′

update dbo.State set LastModified = GETUTCDATE()
where State.LastModified > ‘12-31-2009′

From this change on state changes will restart to flow in.

Issues: monitor needs to be reset or you must wait for the first state change for them to be updated or you could use Marius’ utility Tool- OpsMgr 2007 – RuntimeHealthExplorer or you could use a powershell script to reset all the postdated monitors. The basic statements need to be:

$obj = Get-MonitoringObject -id:<<basemanagedentityid from previous queries>>

$obj.ResetMonitoringState([guid]’<<monitorid from previous queries’)

Last Warning: if you reset the monitor from UI and from a Watcher view then the new healthstate won’t rollup (at least in my env). For example if you reset any unit monitor related to a HealthService starting from the Health Explorer for the related HealthServiceWatcher, the unit monitor will reset but the new status won’t rollup. If you do the same reset from the HealthService Health Explorer view it will rollup.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

If you’re still running OpsMgr 2007 SP1 you definitely need to apply the rollup package has been released yesterday (finally): Update Rollup for Operations Manager 2007 Service Pack 1 (KB971541).

It should set an end to the patching blues I complained so many times about.

Hopefully a similar rollup will be delivered for R2 sooner than later from what I know. Some issues fixed in SP1 are still present in R2.

In the and my advice is if you’re still on SP1 it is still a good idea to move to R2.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

.

While I was trying to understand why my cluster nodes won’t dismiss I dug a little more inside cluster monitoring with opsmgr. As usual I have  no access to source code so I can go wrong on some assumptions.

The core logic behind cluster discovery and management is coded (natively) in mommodules.dll.

The dll exports:

• ClusterGroupStateChange the name gives us some clues
• ClusterDiscovery this one is used by the discovery workflow

Basically every cluster node discovers every resource group (Virtual Server) in the cluster and establish a relationship of type HealthServiceShouldManageEntity. This tells OpsMgr infrastructure to route the workflows for any Virtual Server to every cluster node. In this scenario every cluster node receives all the workflows even for VS it is not owning at the moment. Obviously we just want the owning node to monitor the proper VS. Without some custom logic here we would have an issue, in fact the agent has builtin logic to understand which VS it is supposed to manage (i.e. it is owning). On the passive nodes the workflows get unloaded. Second issue to face is the management of VS failover (i.e. when a resource group changes owning node). From what I understand the agent uses the ClusterGroupStateChange to understand when a VS changes ownership, I measured a 60” maximum delay from resource group failover to workflows reload on the proper node. So far so good, the agent (as we expect) is able to manage the VS where ever it is. I had a couple of cases where this was not working properly on SP1 and it resolved restarting the health service.

One more thing to add, the VSs are managed by the healthservice as a proxied systems, this has an important implication if you’re a MP author: all the workflows you want to execute against a VS must be tagged as remotable=”true”

If you miss this important requirement you’ll get event id 1207 after the agent reloads the workflows

Event Type: Warning

Event Source: HealthService

Event Category: Health Service

Event ID: 1207

Date: 10/17/2009

Time: 3:01:23 PM

User: N/A

Computer: ARES1

Description:

Rule/Monitor "QND.Test.Cluster.LogEvent" running for remote instance "XXXX.progel.org" with id:"{A16D2CDA-378D-E9AC-7913-404A9999BEEE}" will be disabled as it is not remotable. Management group "Progel Labs".

This is it, straightforward but useful if you need to debug issues on your clustered agents.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

The Progel Windows Scheduled Task MP is now available from the MP catalog. We have received a single alert it has issues on extended character set locales (i.e. Traditional Chinese) but with too few details at the moment to assess if it’s our own bug or something external to us. If you manage to try the MP and encounter any issue drop an email to sst@progel.it.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

Disk performance reporting and trending is probably the most difficult part of performance troubleshooting and capacity planning and trending. During the years I used several counters just to understand none of them can give a synthetic and accurate answer on disk performance. Ever tried to user % idle time or % disk time? Or Avg queue length if that matters. In the last few years I standardized on Avg Disk Sec / read / Write / Transfer as a single indicator of disk responsiveness. Generally I take 20ms to 30ms as a threshold should not  be exceeded on average.

However, even these counters are prone to errors (I don’t know when the OS team will change the perf counters architecture, but it will never be too early). First of all you must be aware of the following issues that arise with virtualized Windows 2003 servers with more than one core:

• http://blogs.technet.com/yongrhee/archive/2009/08/04/disk-performance-counters-are-high-are-you-running-into-the-usepmtimer-rtsc-time-drift-issue.aspx
• Programs that use the QueryPerformanceCounter function may perform poorly in Windows Server 2000, in Windows Server 2003, and in Windows XP

But issues are there to hit on Windows 2008 as well. Look at the following table that reports Avg Disk sec / Transfer on a Windows 2008 server:

As you can see, among “normal” values (0,xxx , is the decimal separator in Italy), we have clearly bad ones. It is obvious the server is not taking 93” or 3139” seconds (little less than 1 hour) to execute an I/O on average. The presence of these bad values can wrack havoc your reporting experience, the average of the above values is 285”, now it is clear this cannot be the case.

I didn’t find a root cause for this behavior. I can observe it on several Windows 2008 servers with a predominance of hyper-v hosts, it can be hardware related or just a bug in the perf counter (immo both), in any case your reports are doomed.

The only thing I can advice on is to change your SQL query to filter out obviously bad values. For example filtering out response time above 2 seconds, changes my average on the period to 0.029 seconds or 29 msec that denotes a fairy busy storage subsystem.

If I manage to find more info on this issue I’ll keep you posted, in the meantime take your reports on disk response time with a grain of salt.

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

I think it’s time to wrap the various answers we can find on cluster monitoring and decommissioning in R2.

Failover clustering is an high availability solution and my quote is “if you want to keep it HA you need to monitor it”. So we all expect OpsMgr to do an egregious job in monitoring failover clusters. In some respect it does but you must be aware of glitches that are still around.

I observed at least 3 issues on failover cluster monitoring:

1. abnormal CPU usage on the cluster node and on the controlling management server
2. simply put you cannot decommission a cluster in a supported way
3. the agentless managed via in the administration space of the monitoring console is just useless
4. cluster discovery is noisy

Abnormal CPU usage is caused by internal tasks taking the wrong route. In some cases the config service on the RMS thinks resources are on the wrong node (i.e a node that’s not owning the resources) and sends health recalculation tasks to that node. In this cases you’ll see the following:

• healthservice edb grows we had examples of > 3GB, this grow is caused by tasks queuing in
• healthservice.exe on the affected node consumes up to one cpu core
• healthservice.exe on the primary management server registers an increased cpu usage

We’re waiting for a fix from PSS, at the time the only work around is a group failover, but hey I’m supposed to run critical applications on my clusters a failover cannot be considered a workaround. Btw things can return bad after a while, so I would define this a temporarily rag that will break again sooner or later.

Decommissioning is another issue, the short story is that you cannot remove agents from cluster nodes, but you cannot remove the agents from your console. In other words you have zombies in console.

There’s an *unsupported* workaround here http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!163.entry and a discussion thread here http://social.technet.microsoft.com/Forums/en-US/operationsmanagergeneral/thread/10ffee08-b875-47af-b788-db07dbfa1b56.

See my own unsupported way later in this post, I’m not comfy with the workaround I cite before.

I want to say that the following rapid publishing KB just doesn’t work in my case (and from what I found it should never work): OpsMgr 2007: How to decommission a cluster monitored by System Center Operations Manager 2007

I want to add that the release notes are not clear as well:

“You cannot uninstall or delete an agent from a node in a cluster

When you try to uninstall or delete an agent from a node in cluster, the following error message is displayed:

Agent is managing other devices and cannot be uninstalled. Please resolve this issue via Agentless managed view in Administration prior to attempting uninstall again.

Notice that the agent can be uninstalled from the node that is agentlessly managing the virtual servers. However, the agent cannot be uninstalled from the node that is managing the virtual servers.

Workaround: None at this time.” http://technet.microsoft.com/en-us/library/dd827187.aspx

The agentless managed view is useless in terms that the result’s you’re seeing is just unpredictable and the change proxy action won’t work. This is all related on how the agentless management works for the UI.

An agentless managed Windows Computer (this is the class) is defined as a Windows Computer managed by an healthservice on another Windows Computer (using the HealthServiceShouldManageEntity relationship), this is the query the SDK runs against your live db where the guid is the id for Microsoft.Windows.Computer:

exec sp_executesql N’– AgentlessManagedDevicesByType <ManagedTypeId>

SELECT [T].[Id], [T].[Name], [T].[Path], [T].[FullName], [T].[DisplayName], [T].[IsManaged], [T].[IsDeleted], [T].[LastModified], [T].[TypedManagedEntityId], [T].[MonitoringClassId], [T].[TypedMonitoringObjectIsDeleted], [T].[HealthState], [T].[StateLastModified], [T].[IsAvailable], [T].[AvailabilityLastModified], [T].[InMaintenanceMode], [T].[MaintenanceModeLastModified], [PXH].[BaseManagedEntityId] AS [HealthServiceId], [PXH].[DisplayName] AS [ProxyAgentPrincipalName] FROM dbo.ManagedEntityGenericView AS T

INNER JOIN dbo.BaseManagedEntity AS BME 
    ON

            BME.[BaseManagedEntityId] = T.[Id]

            AND BME.[BaseManagedTypeId] = @ManagedTypeId

INNER JOIN dbo.Relationship AS R 
    ON R.[TargetEntityId] = T.[Id]

INNER JOIN dbo.BaseManagedEntity AS PXH 
    ON PXH.[BaseManagedEntityId] = R.[SourceEntityId]

WHERE ((

            T.[IsDeleted] = 0 AND T.[TypedMonitoringObjectIsDeleted] = 0 AND R.[IsDeleted] = 0 AND

            R.[RelationshipTypeId] = dbo.fn_ManagedTypeId_MicrosoftSystemCenterHealthServiceShouldManageEntity()

          ))’,N’@ManagedTypeId uniqueidentifier’,@ManagedTypeId=’EA99500D-8D52-FC52-B5A5-10DCD1E9D2BD’

What happens for cluster is that we have two HealthServiceShouldManageEntity relationships discovered, one from an internal discovery source (in my case with guid “85AB926D-6E0F-4B36-A951-77CCD4399681”), I would call it the standard one or the discovery source the sdk call SetProxyAgent works with, the other is discovered by the cluster management pack by the discovery rule Microsoft.Windows.Cluster.Classes.Discovery”. This discovery is implemented as a native COM in MOMModules.dll.

Then you must add that every cluster node discovers the entire hierarchy of cluster resources (huge load on every node for complex cluster implementations), so every cluster virtual server is managed by every single node. Let’s take a simple example a basic two node cluster (nodeA and nodeB) with just the cluster virtual server (CLUSTER). The Microsoft.Windows.Computer CLUSTER is discovered by both nodes so that we have that nodeA and nodeB have a HealthServiceShouldManageEntity relationship with CLUSTER with source  Microsoft.Windows.Cluster.Classes.Discovery and another one with the standard discovery source.

  So what happens here is that every cluster node ShouldManage every cluster Virtual Server, for this reason the proxy assignment you see in console is just unpredictable. The selection is based on HealthServiceShouldManageEntity relationship which returns one row for each cluster node for each cluster virtual server.

Using SetAgentProxy from SDK (or change proxy in UI) is useless because it resets just the standard discoverysource and not the Microsoft.Windows.Cluster.Classes.Discovery one.

The root cause of your inhability to delete the virtual servers first and the the cluster nodes then, is related to this discovered relationship. So if you delete the relationship you’ll find your way home.

Here is a SQL snippet that given the cluster node name (FQDN) will delete all the relationships to cluster virtual servers:

declare @nodeHS nvarchar(255)

Set @nodeHS=N’Microsoft.SystemCenter.HealthService:FQDN’

DECLARE Rel_Cursor CURSOR FOR

(SELECT [RelationshipGenericView].[Id]

FROM dbo.RelationshipGenericView

WHERE ((RelationshipGenericView.[MonitoringRelationshipClassId] =  dbo.fn_ManagedTypeId_MicrosoftSystemCenterHealthServiceShouldManageEntity()) AND (((dbo.[RelationshipGenericView].[IsDeleted] = 0))))

AND ([RelationshipGenericView].[SourceMonitoringObjectId]

    IN (select BaseManagedEntityId from BaseManagedEntity where FullName =@NodeHS)))

OPEN Rel_Cursor;

declare @relId uniqueidentifier

declare @discoSource uniqueidentifier

declare @now datetime

set @now = GETUTCDATE()

FETCH NEXT FROM Rel_Cursor INTO @RelId;

WHILE @@FETCH_STATUS = 0

BEGIN

    SELECT @discoSource=DSTR.[DiscoverySourceId]

    –SELECT *

    FROM  dbo.[DiscoverySourceToRelationship] DSTR

        inner join dbo.[DiscoverySource] DS on DS.DiscoverySourceId = DSTR.DiscoverySourceId

        inner join Discovery on Discovery.DiscoveryId = DS.DiscoveryRuleId

                   WHERE [DiscoveryName] = ‘Microsoft.Windows.Cluster.Classes.Discovery’

                   AND [RelationshipId]=@relId

                   AND DSTR.[IsDeleted] = 0

    exec dbo.p_RemoveRelationshipFromDiscoverySourceScope

    @RelationshipId=@relId,

    @DiscoverySourceId=@discoSource,@TimeGenerated=@now                  
    FETCH NEXT FROM Rel_Cursor INTO @RelId;

END;

CLOSE Rel_Cursor;

DEALLOCATE Rel_Cursor;

After executing this SQL statement the cluster node and virtual servers can be deleted from the Operations Console. Obviously this is totally unsupported. So do this at your own risk, but if you followed my analysis it is clear it should be safe.

Cluster discovery is noisy, this means too much 21025 events (i.e. agents reloads) and the way it is performed on every node can lead to serious impact and cluster nodes. While I was about to post the detail of the noisy discovery (starting with mac address mismatch with OS discovery) a new cluster MP has been released (6.0.6720.0), this one promises to solve a lot of the issue I detected in previous versions. So this one is strongly recommended. btw it adds support for Windows Server 2008 R2 failover clusters (yes CSV included).

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

I encountered this problem while preparing a demo consisting in a migration from Windows XP to Windows 7. I was using USMT 4, MDT 2010 and SCCM 2007 SP2 RC. In my task sequence I used hard-links to capture user data from a Windows XP installation and then to restore data back after applying a Windows 7 Image previously created. The migration process completed successfully but no system component settings were migrated. I looked at the scanstate.log created in %systemroot%\system32\ccm\logs\SMSTSLog to see if something went wrong and I found the following line :

2009-10-17 17:19:44, Info [0x000000] Downlevel Manifests folder is not present. System component settings will not be gathered.

The error is related to a not present manifest folder, so I used FileMon lo verify in which location scanstate was looking :

Scanstate looked for that folder in %Systemroot%\system32, same location as the “Current Directory” of the process (as shown by Process Explorer).

By doing a simple search with Google I’ve found that I’m not alone and that other people experienced the same issue (http://systemcenterideas.com/2009/09/usmt-issues-with-mdt-2010). The fix proposed in that blog does not apply to me because I use a “Capture Task” and not a script to execute scanstate.

So while waiting to see if it will be fixed in RTM (I hope so) I needed a workaround.

I decided to create a wrapper that :

• executes a renamed scanstate.exe as a child process.
• passes the same parameters received from the command line to the child process
• passes the Application path as the Current Directory of the child process
• returns the child process exit code.

I used :

• GetCommandLine : to retrieve parameters passed to the wrapper
• GetModuleFileName : to get the application path (after eliminating the application file name)
• CreateProcess : to call the renamed scanstate and to pass the application folder as the Default Directory
• GetExitCodeProcess : to retrieve the child process exit code and to pass it to the task sequence.

I renamed scanstate.exe to _scanstate.exe and I putted my wrapper in the USMT x86 folder, naming it scanstate.exe.

As it can be seen in the previous picture, my wrapper calls the real scanstate application (named _scanstate.exe) as a child process and forces the “Current Directory” to be the same where the wrapper is located. With the correct “Default directory” _scanstate.exe is able to access the manifest folder.

This is a sort of hack, it is not supported an probably it is not the best solution to this problem, but in this case I’m running an RC version of SCCM 2007 SP2 and I’m only interested in have the demo working in the right way. I really hope that this will be fixed in the RTM version of SP2 or that if there is a supported alternative solution to this problem, that will be published soon (I know that I can copy manifest folder to the system32 before calling scanstate, but I don’t like to do that).

– Fabrizio

This posting is provided "AS IS" with no warranties, and confers no rights.

I read several post about issues agents are facing even with R2. Yes R2 has still issues (clustering anyone?), but before pointing your finger at OpsMgr you should consider that a monitoring agent uses interfaces that are not normally used and this can lead to “new” bug discovered in OS or application components. So it’s not an agent issue, but the bug rises up only after agent installation. Bottom line: monitoring is never for free even agentlessly.

Before pointing your finger at OpsMgr this is our recommended fix list.

On every OS

On Windows Server 2008

• Service Pack 2

On Windows Server 2003

• windows scripting host 5.7 on Windows 2003 (and Windows 2000)
• KB 956523 on Windows 2003 to address a memory leak in WMI
• KB 931320 another issue with WMI on Windows 2003
• KB 943071 issue with event provider in managed code and WMI on Windows 2003
• KB 933061 it fixes several issues in WMI on Windows 2003, it is of great help with WMI issues even if it won’t resolve them all

On Biztalk 2007 servers (very noisy MP btw)

• you *must* follow what’s reported here http://msdn.microsoft.com/en-us/library/ee290753(BTS.10).aspx. You’ll find specific fixes to be applied other than the ones I report in this short list.

I’ll try to keep this post up to date with any new fix we’ll consider useful for agent health.

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.