SCOM patching blues
This one is an old favorite of mine. I tried to explain to the team (even with flames on Connect) the importance of proper patching, alas with no luck. They had a serious issue where windows installer returned successful while patching , in reality, didn’t take place. A lot of QFEs have been silently re-released to address the issue. Moreover the patchlist field in Health Service Class (thanks to QFE 958253) now reports only the KB numbers, so now it’s possible to track QFE applied on each HealthService.
Alas the blues is for from over, due to the sequencing error in old QFE I discovered to have plenty of agents not up to date, even if the QFE is “apparently” installed. The only way to check for this issue is to get the file version of every single affected file on every single agent. To address this I developed a quick MP that collects for me this data and adds a monitor for required version for each module. If anyone is interested in this one let me know and I will check if I can share it.
KB articles do not report superseding of old QFE with new ones, and KB 956689 (recently re-released) is still a mystery to me, it states that just MOMNetworkModules.dll is being affected, but if you open the patch file with Orca you’ll find several files get modified without even updating the version number.
So, after trying for the last 4 months to address the issue with PSS and the product team, I finally decided to do my own checks and to build my list of must have hotfixes (here I’m referring just to agent QFE, not to RSM/MS or MP specific QFEs). Since this took a lot of work, I’d like to share this list with the community:
- KB 954049
- KB 954903
- KB 956172
- KB 956689 (still doubtful, since the MOMNetworkModules.dll has been superseded by 957511)
- KB 957511
this means that your AgentManagement directory on every MS should be like this:
Once again hope this will help.
Update: I’m authorized to share a stripped down version of the MP I’ve been referring to Progel.HSVersion.Public.xml.
– Daniele
This posting is provided "AS IS" with no warranties, and confers no rights.
Great posting Daniele.
This is helping me with administration of several environments. Maybe some sort of excel should be created to sum up fix, comments, related links (like this site and Kevin Holmans blog).
Also this would be great opportunity for the OpsMgr team to step up and admit okay there are these bad consequences of applying these certain patches and also explaining what they are planning to do in the future.
It is always easier to discuss with customer when you have also the official opinion of manufacturer so if something goes wrong, you have a backup why you did how you did.
Keep up the good posts!
Tero Ilenius
March 19, 2009 at 11:56 am
I don’t know why it includes other files too, and I had not noticed that myself.
It might be just that the code did not change for those other files, but they got re-compiled as they needed re-linking ?
MomNetworkModules.dll seems to be a COM DLL (it exposes DLLRegisterServer)… while HealthServiceRuntime.dll is a native, win32 DLL…
Again, it is just my *idea* of this at this point, not necessarily the real thing/answer.
I will try to ask that to the developers… or maybe they are already reading us…
Daniele Muscetta
March 14, 2009 at 2:19 pm
Daniele, I got an answer by one of our test managers, and he said:
[...] Only the binaries with higher version are replaced. If file is unversioned then checksum is used to determine if it should be updated. Just because Orca shows file sequence or attributes have changed does not mean it will be updated [...]
Daniele Muscetta
March 23, 2009 at 5:30 pm
Hi Daniele,
obviously you’re right, but the file size (i.e the checksum) is changed not just the sequence, and indeed the files are updated on the file system.
Daniele Grandini
March 23, 2009 at 6:48 pm
also check http://msdn.microsoft.com/en-us/library/aa367835(VS.85).aspx
Daniele Muscetta
March 23, 2009 at 5:31 pm
Following up from Blake request and other private ones, I developed a stripped down MP (http://cid-558ec647eef17f8d.skydrive.live.com/self.aspx/.Public/Sample%20MPs/Progel.HSVersion.Public.xml) that complies with my company policies. The MP just discovers the modules (i.e. dlls) into the class “Health Service Module”. Alas I cannot share the monitoring stuff.
- Daniele
Daniele Grandini
March 14, 2009 at 12:30 pm
excellent suggestion. everything works perfectly
Vincenzo Gaudio
March 13, 2009 at 3:21 pm
the list above is currently pretty much the same list we (in Premier Field Engineer) install on each customer we visit… also check out Kevin’s list: http://blogs.technet.com/kevinholman/archive/2009/01/27/which-hotfixes-should-i-apply.aspx
and no, since 957511 replaces the other version of momnetworkmodules.dll, that is the only one that needs to be installed for that file – not the previous/older one. latest build of the same file “wins”, and includes the previous fixes – as they are built incrementally.
We keep giving this kind of feedback internally to the Product Team too… you’re not alone.
958253 has been a first attempt to make this better… I suppose it takes time.
Daniele Muscetta
March 12, 2009 at 4:14 pm
Hi Daniele, glad to hear you share the same patch list, but still 956689 updates much more files than MOMNetworkModules.dll as my Orca snapshot testifies. SO, what’s going wrong here? For example HealthServiceRuntime.dll gets updated by 956689 (binaries and date are updated) but still version is .0.
Daniele Grandini
March 12, 2009 at 4:36 pm
I would appreciate that MP if you can share. Love this write up, thanks dude! mengottohotmail.com (email addr for the mp if you can send)
Blake Mengotto
March 11, 2009 at 6:53 pm