SCOM patching blues


This one is an old favorite of mine. I tried to explain to the team (even with flames on Connect) the importance of proper patching, alas with no luck. They had a serious issue where windows installer returned successful while patching , in reality, didn’t take place. A lot of QFEs have been silently re-released to address the issue. Moreover the patchlist field in Health Service Class (thanks to QFE 958253) now reports only the KB numbers, so now it’s possible to track QFE applied on each HealthService.

Alas the blues is for from over, due to the sequencing error in old QFE I discovered to have plenty of agents not up to date, even if the QFE is “apparently” installed. The only way to check for this issue is to get the file version of every single affected file on every single agent. To address this I developed a quick MP that collects for me this data and adds a monitor for required version for each module. If anyone is interested in this one let me know and I will check if I can share it.

image

KB articles do not report superseding of old QFE with new ones, and KB 956689 (recently re-released) is still a mystery to me, it states that just MOMNetworkModules.dll is being affected, but if you open the patch file with Orca you’ll find several files get modified without even updating the version number.

image

So, after trying for the last 4 months to address the issue with PSS and the product team, I finally decided to do my own checks and to build my list of must have hotfixes (here I’m referring just to agent QFE, not to RSM/MS or MP specific QFEs). Since this took a lot of work, I’d like to share this list with the community:

  • KB 954049
  • KB 954903
  • KB 956172
  • KB 956689 (still doubtful, since the MOMNetworkModules.dll has been superseded by 957511)
  • KB 957511

this means that your AgentManagement directory on every MS should be like this:

image

Once again hope this will help.

Update: I’m authorized to share a stripped down version of the MP I’ve been referring to Progel.HSVersion.Public.xml.

Technorati Tags: ,,

– Daniele

This posting is provided "AS IS" with no warranties, and confers no rights.

Advertisements
  1. #1 by Tero Ilenius on March 19, 2009 - 11:56 am

    Great posting Daniele.

    This is helping me with administration of several environments. Maybe some sort of excel should be created to sum up fix, comments, related links (like this site and Kevin Holmans blog).

    Also this would be great opportunity for the OpsMgr team to step up and admit okay there are these bad consequences of applying these certain patches and also explaining what they are planning to do in the future.

    It is always easier to discuss with customer when you have also the official opinion of manufacturer so if something goes wrong, you have a backup why you did how you did.

    Keep up the good posts!

  2. #2 by Daniele Muscetta on March 14, 2009 - 2:19 pm

    I don’t know why it includes other files too, and I had not noticed that myself.
    It might be just that the code did not change for those other files, but they got re-compiled as they needed re-linking ?
    MomNetworkModules.dll seems to be a COM DLL (it exposes DLLRegisterServer)… while HealthServiceRuntime.dll is a native, win32 DLL…
    Again, it is just my *idea* of this at this point, not necessarily the real thing/answer.
    I will try to ask that to the developers… or maybe they are already reading us…

    • #3 by Daniele Muscetta on March 23, 2009 - 5:30 pm

      Daniele, I got an answer by one of our test managers, and he said:

      […] Only the binaries with higher version are replaced. If file is unversioned then checksum is used to determine if it should be updated. Just because Orca shows file sequence or attributes have changed does not mean it will be updated […]

      • #4 by Daniele Grandini on March 23, 2009 - 6:48 pm

        Hi Daniele,
        obviously you’re right, but the file size (i.e the checksum) is changed not just the sequence, and indeed the files are updated on the file system.

    • #5 by Daniele Muscetta on March 23, 2009 - 5:31 pm

  3. #6 by Daniele Grandini on March 14, 2009 - 12:30 pm

    Following up from Blake request and other private ones, I developed a stripped down MP (http://cid-558ec647eef17f8d.skydrive.live.com/self.aspx/.Public/Sample%20MPs/Progel.HSVersion.Public.xml) that complies with my company policies. The MP just discovers the modules (i.e. dlls) into the class “Health Service Module”. Alas I cannot share the monitoring stuff.

    – Daniele

  4. #7 by Vincenzo Gaudio on March 13, 2009 - 3:21 pm

    excellent suggestion. everything works perfectly

  5. #8 by Daniele Muscetta on March 12, 2009 - 4:14 pm

    the list above is currently pretty much the same list we (in Premier Field Engineer) install on each customer we visit… also check out Kevin’s list: http://blogs.technet.com/kevinholman/archive/2009/01/27/which-hotfixes-should-i-apply.aspx

    and no, since 957511 replaces the other version of momnetworkmodules.dll, that is the only one that needs to be installed for that file – not the previous/older one. latest build of the same file “wins”, and includes the previous fixes – as they are built incrementally.

    We keep giving this kind of feedback internally to the Product Team too… you’re not alone.
    958253 has been a first attempt to make this better… I suppose it takes time.

    • #9 by Daniele Grandini on March 12, 2009 - 4:36 pm

      Hi Daniele, glad to hear you share the same patch list, but still 956689 updates much more files than MOMNetworkModules.dll as my Orca snapshot testifies. SO, what’s going wrong here? For example HealthServiceRuntime.dll gets updated by 956689 (binaries and date are updated) but still version is .0.

  6. #10 by Blake Mengotto on March 11, 2009 - 6:53 pm

    I would appreciate that MP if you can share. Love this write up, thanks dude! mengottohotmail.com (email addr for the mp if you can send)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: