How to get noisy discovery rules


Frequently changing properties and objects can be one of the causes of poor OpsMgr performance. In RC bits we saw optimizations on this topic, but still you must be aware that every configuration reload on the agents (21025 events but internal tasks as well) is going to tax your CPU. How much? It depends on MPs number, complexity and CPU power. I must admit I’m a little paranoid on agent health and resource utilization, but I do not want the monitoring infrastructure have a negative impact on business processes. Monitoring is great if it prevents down time not if it causes it, doesn’t it?

So, how can I check for bad MP behaviors. In a steady system newly discovered objects should be 0 or a few, discovered properties should register few changes. So the first question we must ask is which newly discovered objects are we getting and from which discovery rules. The second question is which properties are changing frequently and which are the related discovery rules. Initially I though to develop a powershell script, but I quickly changed my mind:

  1. powershell uses the SDK it can access just the live DB that typically contains a few hours or at most a few days or data
  2. I’d like to report on these changes so that I can have a weekly report in my inbox

So I turned to SQL queries against the data warehouse. I got a fairly accurate query for both questions. The only topic I must highlight is that the connection between the discovered objects and properties and the object or property itself is calculated based on the configuration part of the discovery rule:

<Discovery ID="Microsoft.SQLServer.2008.DBEngineDiscoveryRule.Server" Enabled="true" Target="Windows!Microsoft.Windows.Server.Computer" ConfirmDelivery="false" Remotable="true" Priority="Normal">
  <Category>Discovery</Category>
  <DiscoveryTypes>
    <DiscoveryClass TypeID="Microsoft.SQLServer.2008.DBEngine">
      <Property TypeID="SQL!Microsoft.SQLServer.ServerRole" PropertyID="InstanceName" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="ConnectionString" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="ServiceName" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="ServiceClusterName" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="FullTextSearchServiceName" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="FullTextSearchServiceClusterName" />
      <Property TypeID="SQL!Microsoft.SQLServer.DBEngine" PropertyID="Version" />

but the <DiscoveryTypes> fragment of the discovery rules is not enforced, so we can have missing discovery rules. ON the other side the same class and property can be discovered by multiple rules, so we can a few more hits. But in any case this level of approximation is neglectable.

Discovered objects in the last 4 hours:

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As ‘TargetTypeSystemName’, MET1.ManagedEntityTypeDefaultName ‘TargetTypeDefaultName’,
ME.Path, ME.Name,
ME.DWCreatedDateTime
from dbo.vManagedEntity ME
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
    AND CAST(DefinitionXml.query(‘data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)’) AS nvarchar(max)) like ‘%’+MET.ManagedEntityTypeSystemName+’%’
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ME.DWCreatedDateTime > dateadd(hh,-4,getutcdate())

Modified properties in the last 4 hours:

select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As ‘TargetTypeSystemName’, MET1.ManagedEntityTypeDefaultName ‘TargetTypeDefaultName’,
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
    AND CAST(DefinitionXml.query(‘data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)’) AS nvarchar(max)) like ‘%’+MET.ManagedEntityTypeSystemName+’%’
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-4,getutcdate())

Top discovery rule in the last 4 hours:

select ManagedEntityTypeSystemName, DiscoverySystemName, count(*) As ‘Changes’
from
(select distinct
MP.ManagementPackSystemName,
MET.ManagedEntityTypeSystemName,
PropertySystemName,
D.DiscoverySystemName, D.DiscoveryDefaultName,
MET1.ManagedEntityTypeSystemName As ‘TargetTypeSystemName’, MET1.ManagedEntityTypeDefaultName ‘TargetTypeDefaultName’,
ME.Path, ME.Name,
C.OldValue, C.NewValue, C.ChangeDateTime
from dbo.vManagedEntityPropertyChange C
inner join dbo.vManagedEntity ME on ME.ManagedEntityRowId=C.ManagedEntityRowId
inner join dbo.vManagedEntityTypeProperty METP on METP.PropertyGuid=C.PropertyGuid
inner join dbo.vManagedEntityType MET on MET.ManagedEntityTypeRowId=ME.ManagedEntityTypeRowId
inner join dbo.vManagementPack MP on MP.ManagementPackRowId=MET.ManagementPackRowId
inner join dbo.vManagementPackVersion MPV on MPV.ManagementPackRowId=MP.ManagementPackRowId
left join dbo.vDiscoveryManagementPackVersion DMP on DMP.ManagementPackVersionRowId=MPV.ManagementPackVersionRowId
    AND CAST(DefinitionXml.query(‘data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)’) AS nvarchar(max)) like ‘%’+MET.ManagedEntityTypeSystemName+’%’
left join dbo.vManagedEntityType MET1 on MET1.ManagedEntityTypeRowId=DMP.TargetManagedEntityTypeRowId
left join dbo.vDiscovery D on D.DiscoveryRowId=DMP.DiscoveryRowId
where ChangeDateTime > dateadd(hh,-4,getutcdate())
) As #T
group by ManagedEntityTypeSystemName, DiscoverySystemName
order by count(*) DESC

and this is a sample output from my environment… I must work on the DPM MP

image

- Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

About these ads
  1. #1 by Sylvester on April 30, 2010 - 4:07 pm

    Nice article, but there is a little thing in the following condition:

    CAST(DefinitionXml.query(‘data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)’) AS nvarchar(max)) like ‘%’+MET.ManagedEntityTypeSystemName+’%’

    The problem is that it selects all discovery types that begin with ManagedEntityTypeSystemName. In our case it adds several discovery rules to the result table with invalid target type.

    Thanks,
    Sylvester

    • #2 by Daniele Grandini on April 30, 2010 - 5:14 pm

      Hi Sylvester,
      you’re right I’m using a loose check because I’m not sure of the decoration of the TypeID property once it gets in the database, I’m actually searching for any Discovery rule for which a DiscoveryClass whose typeid contains (not just starts) the ME Type System Name has been defined (sadly there’s no enforcement in the MP schema to declare the classes you’re discovering). You can try to change the check with an =. Let me know if it works, I fear no.

  2. #3 by Michael Skov on March 23, 2010 - 1:38 pm

    Hi

    Look cool, but i am not able to get any information. I am running the query against my OpsMgr DW, but getting this:

    Msg 102, Level 15, State 1, Line 1
    Incorrect syntax near ‘‘’.
    Msg 102, Level 15, State 1, Line 8
    Incorrect syntax near ‘‘’.

    Thanks

    Regards
    Michael

    • #4 by Daniele Grandini on March 24, 2010 - 11:59 am

      hi Michael, this is due to the formatting editor I’m using, I will update the post in the next few hours to change the backward apex.
      Thanks
      Daniele

  3. #5 by Jonathan Almquist on June 10, 2009 - 5:35 am

    Daniele,

    This is excellent work. As you surely know, many of your works are used and sometimes “reposted” on other blogs (including TN). I want to thank you for sharing your invaluable insights and knowledge, and give you the credit you deserve.

    -Jonathan

    • #6 by Daniele Grandini on June 11, 2009 - 9:26 am

      Jonathan I wish to thank you for this comment. It’s great to know my post are somewhat useful, this encourages me to continue with this blog. I’d really like to do more, I have some many insights to share but my time is limited…

  4. #7 by Steve Burkett on May 26, 2009 - 1:46 pm

    Hmm.. Trying to get the ‘Top discovery rule in the last 4 hours’ query working here and am getting an error:

    Msg 208, Level 16, State 1, Line 1
    Invalid object name ‘dbo.vManagedEntityPropertyChange’.

    Any ideas?

    • #8 by Daniele Grandini on May 27, 2009 - 5:47 pm

      Uhm are you running the query against the Data Warehouse database? This view is a standard one (at least in R2 I didn’t check in SP1, but I think it was there as well). If you’re running the query against the DW and you don’t have the view, well I think your DW is not so healthy, something must have gone wrong during installation. Just my first though.

      • #9 by keriaks on October 19, 2011 - 6:10 pm

        I get the same error what Steve mentioned here “Invalid object name ‘dbo.vManagedEntityPropertyChange’.” we Have SCOM 2007 R2. I do not see that view in Ops DB . Please suggest. Thank you.

      • #10 by Daniele Grandini on October 20, 2011 - 8:31 am

        Hi keriacs,
        you should find the view in the Data warehouse database, it is standard view I guess you have it.

        – Daniele

      • #11 by keriaks on October 20, 2011 - 2:50 pm

        This works. Thank you.

  5. #12 by dani3l3 on May 24, 2009 - 7:18 am

    We have been doing something similar in the OpsMgr Health Check for a long time, already. The problem with noisy discoveries is that you have limited options about how to tackle those. They are more an indicator of poor MP quality than of user configuration. When you spot a discovery is “noisy” and updates properties too often… what can you really do, then, if it is a vendor MP?

    • #13 by Daniele Grandini on May 26, 2009 - 10:29 am

      Hi Daniele,
      I don’t know what you’re doing in the OpsMgr Health Check, what we do is:
      1) getting evidence of bad MPs
      2) first try: override the discovery frequency to at least 4 hours
      3) second try: disable the discovery rule and build our own. This is time consuming ’cause you need to check which discovered properties are used in monitoring and which not

  1. What is config churn? - Kevin Holman's OpsMgr Blog - Site Home - TechNet Blogs
  2. 2010 in review « Quae Nocent Docent
  3. Troubleshooting 21025 events – wrap up « Quaue Nocent Docent
  4. Troubleshooting 21025 events – Part 1 evidence « Quaue Nocent Docent
  5. Steve Rachui's Manageability blog - ConfigMgr/OpsMgr : Is your RMS updating configuration too frequently?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 330 other followers

%d bloggers like this: