Troubleshooting 21025 events – part 2 the RMS case


As Daniele pointed out in the article (Troubleshooting 21025 events – Part 1 evidence) 21025 events on the RMS are bad. In one of our SCOM 2007 R2 environment we observed an high number of those events on the RSM and an impact on the scalability of that infrastructure. So I decided to try to investigate the cause of this unusual number of 21025 events on that specific RSM. I started looking at OpsMgrTrace but with more than 450 agents it was difficult to understand what was going on because of the size of the trace. Fortunately in that environment agents communicate with RMS trough 10 gateways, so i stopped 9 gateways having only 15 agents active during my tests. Looking inside the trace I found the line in which OpsMgr Config Service forces RMS Health Service to Update his configuration :

[ConfigService] [] [Information] :HealthService.HealthServiceState.OnDirty{healthservice_cs1253}( 0000000000C1324E )Logging event 29102 from source OpsMgr Config Service with severity Information and description Configuration state of OpsMgr Health Service {e365284c-48ac-b6d5-0138-0222a107ac46} running on myrms.mydomain.mycountry may be out of date. It should contact OpsMgr Config Service to synchronize its configuration state.

[ConfigService] [] [Information] :StateManager.CollectDirty{statemanager_cs39}( 0000000001F5D7C3 )State=e365284c-48ac-b6d5-0138-0222a107ac46 is dirty

some lines above, I found an entry that seems interesting :

[Bid2Etw_Mom_Dal_1_Trace] [] [Information] :{Information_TextW} Query definition name = ConfigServiceNotification command text = dbo.p_ConfigServiceNotifications

[ConfigService] [] [Information] :ConfigurationChangeSetProvider.UpdateQueryTimestampFromResults{configurationchangesetprovider_cs595}( 0000000003BFFBB0 )Timestamp = 6/28/2009 9:35:49 PM.

[ConfigService] [] [Information] :DatabaseAccessor.NotifyOnChanges{databasenotification_cs329}( 0000000000BF3DF8 )Firing change notification.

I stated looking at the stored procedure dbo.p_ConfigServiceNotifications and IMO it is used to find if something is changed in the environment (Discovery data, ,Relationship, MPs, Rules , Monitor targeted types, Credential Manager Secure Storage, Secure StorageSecure Reference) that need to be notified to the Config Service. I used SQL Profiler to trace the call to this SP and I found that is executed every 30 seconds with parameters similar to the followings :

declare @p3 bit
set @p3=1
exec dbo.p_ConfigServiceNotifications @IsBootStrap=0,@TimeOfLastNotification='2009-06-28 20:44:04:287',@IsDataReliable=@p3 output
select @p3

I used that stored procedure to check if I was able to understand what was changing in that environment that creates an event 21025 on the RMS.

A sample Output of the stored procedure looks like the following :

ManagedTypeId                        TopHostingEntityId                   KeyValue              
------------------------------------ ------------------------------------ ----------------------
D447F983-507B-46C4-ED1B-D467E134A863 1678A4CA-3406-408E-D4E9-D60F0D77A139 pre.lab (DC02.pre.lab)

I checked with the following SQL statement who hosted the Class (it is confirmed that was not the RMS)

select FullName from dbo.BaseManagedEntity  where BaseManagedEntityId = '1678A4CA-3406-408E-D4E9-D60F0D77A139'

FullName
—————————————
Microsoft.Windows.Computer:DC02.pre.lab

I checked with the following SQL statement what class was changed :

select TypeName from dbo.ManagedType where ManagedTypeId = 'D447F983-507B-46C4-ED1B-D467E134A863'

TypeName
———————————-
Microsoft.Windows.DNSServer.Library.Zone

What I found is that some specific classes (DNS Zones, IIS Web Sites, etc) hosted by some servers (not hosted by RSM) generate a 21025 on RMS every time a class property change occurs. I did tests in the past and I observed that a change in a class property generates an event 21025 on the HealtService that hosts the class not on the RMS, so why those specific classes are different from others?

One of the class that generates event 21025 on RSM is a class in a simple custom management pack, so I started to compare that MP with MP that I used in the past to troubleshoot this kind of behavior and I started to remove monitors, rules, etc.. in that MP to understand what is the source of the problem. After some test I removed from that MP a Group that contain al instance of that specific class, without It no 21025 events were generated when a property change of that class occurred.

To verify my hypothesis I created a group in the MP that never generated a 21025 on RSM, and it started to generate them on RMS at every property change on the class contained by the newly created group. I verified the existence of a group that contain DNS Zones and a group that contains IIS Web Sites, so now it is clear why a change on them creates a 21025 on RSM.

In conclusion it seems that if a property of a class changes and the instance of that class is member of group, an event 21025 is created on RMS.

I will update this post with a sample MP that contain a class, a discovery and a group to simulate in lab this behavior.

– Fabrizio

This posting is provided “AS IS” with no warranties, and confers no rights.

Advertisements

,

  1. #1 by PaulD on August 22, 2009 - 12:11 am

    Hi Fabrizion.
    Interesting findings.
    I was pointed to your blog trying to search “p_ConfigServiceNotifications” in google.
    I fished this SP in sql profiler trying to debug event 29106
    The request to synchronize state for OpsMgr Health Service identified by “” failed due to the following exception

  1. Troubleshooting 21025 events – wrap up « Quaue Nocent Docent

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: