The very first step to monitor a generic device is to get a unique identifier for the device itself. This is important since we need to discover the device in order to apply the proper monitoring policies. The first id to check for is sysObjectID (.1.3.6.1.2.1.1.2), the id is returned by OpsMgr discovery itself.

The easy way is to add the generic device IP to our discovery rule and let Operations Manager to perform the basic discovery, if this doesn’t work it’s useless to go forward, the device needs to be discovered by standard Operations manager discovery before we can do anything with it. Once it has been discovered we just need to step in the administration space and, under network devices, select the properties of the device:

 There are obviously other methods to get to this OID for example snmpwalk from SharpSnmpLib can be used (see references in chapter 1), for an SNMP v2 device the syntax is something like this:

snmpget -v=2 -c=public localhost .1.3.6.1.2.1.1.2

where “localhost” must be substituted with the device IP and the community set accordingly, but this is more a troubleshooting technique then something useful at this stage: remember the device must be first discovered by Operations Manager.

Discovering the device

One we have the the device id we can write our discovery. To do this:

1.       We must define a new class derived by System.NetworkManagement.Node. If our device is a Switch we can use the System.NetworkManagement.Switch class as well. What if it is a router, easy System.NetworkManagement.Router.

    <EntityTypes>

      <ClassTypes>

        <!– In this section, modify the attributes based on your needs. Ensure the

             Base attribute matches the application you are modeling.

             More information can be found in the Management Pack Development Kit:

             http://msdn.microsoft.com/en-us/library/ee533867.aspx –>

        <ClassType ID=“QND.MyNetwork.Device“ Accessibility=“Public“ Abstract=“false“ Base=“Network!System.NetworkManagement.Node“ Hosted=“false“ Singleton=“false“ Extension=“false” />

        </ClassType>

</EntityTypes>

2.       Using the System.NetworkManagement.SnmpProbe we must define a new data source module type to feed our discovery rule. To make code reusable I will define a generic datasource that does the match and a specific datasource that perform the actual mapping

  <DataSourceModuleType ID="QND.NetLibrary.MatchOID.DS" Accessibility="Public" Batching="false">

    <Configuration>

      <IncludeSchemaTypes>

        <SchemaType>System!System.ExpressionEvaluatorSchema</SchemaType>

        <SchemaType>Snmp!System.SnmpVarBindsSchema</SchemaType>

      </IncludeSchemaTypes>

      <xsd:element minOccurs="1" name="IntervalSeconds" type="xsd:integer" />

      <xsd:element minOccurs="0" name="SyncTime" type="xsd:string" />

      <xsd:element minOccurs="0" maxOccurs="1" name="NoOfRetries" type="xsd:unsignedInt" default="3" />

      <xsd:element minOccurs="0" maxOccurs="1" name="Timeout " type="xsd:unsignedInt" default="500" />

      <xsd:element minOccurs="0" maxOccurs="1" name="Port" type="xsd:unsignedInt" default="161" />

      <xsd:element name="IP" type="xsd:string" />

      <xsd:element name="CommunityString" type="xsd:string" />

      <xsd:element minOccurs="0" maxOccurs="1" name="Version" type="xsd:string" />

      <xsd:element name="ObjectOID" type="xsd:string" />

    </Configuration>

    <ModuleImplementation Isolation="Any">

      <Composite>

        <MemberModules>

          <DataSource ID="Scheduler" TypeID="System!System.Discovery.Scheduler">

            <Scheduler>

              <SimpleReccuringSchedule>

                <Interval>$Config/IntervalSeconds$</Interval>

                <SyncTime>$Config/SyncTime$</SyncTime>

              </SimpleReccuringSchedule>

              <ExcludeDates />

            </Scheduler>

          </DataSource>

          <ProbeAction ID="Probe" TypeID="Network!System.NetworkManagement.SnmpProbe">

            <Walk>false</Walk>

            <WalkReturnMultipleItems>false</WalkReturnMultipleItems>

            <NoOfRetries>3</NoOfRetries>

            <Timeout>500</Timeout>

            <Port>$Config/Port$</Port>

            <SNMPv3UserName>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/UserName$</SNMPv3UserName>

            <SNMPv3AuthProtocol>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/AuthenticationProtocol$</SNMPv3AuthProtocol>

            <SNMPv3AuthPassword>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/AuthenticationKey$</SNMPv3AuthPassword>

            <SNMPv3PrivProtocol>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/PrivacyProtocol$</SNMPv3PrivProtocol>

            <SNMPv3PrivPassword>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/PrivacyKey$</SNMPv3PrivPassword>

            <SNMPv3ContextName>$RunAs[Name="Network!System.NetworkManagement.SnmpV3.MonitoringAccount"]/ContextName$</SNMPv3ContextName>

            <IsWriteAction>false</IsWriteAction>

            <IP>$Config/IP$</IP>

            <CommunityString>$Config/CommunityString$</CommunityString>

            <Version>$Config/Version$</Version>

            <SnmpVarBinds>

              <SnmpVarBind>

                <OID>.1.3.6.1.2.1.1.2.0</OID>

                <Syntax>0</Syntax>

                <Value VariantType="8" />

              </SnmpVarBind>

            </SnmpVarBinds>

            <OutputOnError>false</OutputOnError>

          </ProbeAction>

          <ConditionDetection ID="FilterOnObjectID" TypeID="System!System.ExpressionFilter">

            <Expression>

              <SimpleExpression>

                <ValueExpression>

                  <XPathQuery Type="String">SnmpVarBinds/SnmpVarBind[OID=".1.3.6.1.2.1.1.2.0"]/Value</XPathQuery>

                </ValueExpression>

                <Operator>Equal</Operator>

                <ValueExpression>

                  <Value Type="String">$Config/ObjectOID$</Value>

                </ValueExpression>

              </SimpleExpression>

            </Expression>

          </ConditionDetection>

        </MemberModules>

        <Composition>

          <Node ID="FilterOnObjectID">

            <Node ID="Probe">

              <Node ID="Scheduler" />

            </Node>

          </Node>

        </Composition>

      </Composite>

    </ModuleImplementation>

    <OutputType>Snmp!System.SnmpData</OutputType>

  </DataSourceModuleType>

  <DataSourceModuleType ID="QND.MyNetwork.Device.Discovery.DS" Accessibility="Internal" Batching="false">

  <Configuration>

    <xsd:element minOccurs="1" name="IntervalSeconds" type="xsd:integer" />

    <xsd:element minOccurs="0" name="SyncTime" type="xsd:string" />

    <xsd:element minOccurs="0" maxOccurs="1" name="NoOfRetries" type="xsd:unsignedInt" default="3" />

    <xsd:element minOccurs="0" maxOccurs="1" name="Timeout " type="xsd:unsignedInt" default="500" />

    <xsd:element minOccurs="0" maxOccurs="1" name="Port" type="xsd:unsignedInt" default="161" />

    <xsd:element name="IP" type="xsd:string" />

    <xsd:element name="CommunityString" type="xsd:string" />

    <xsd:element minOccurs="0" maxOccurs="1" name="Version" type="xsd:string" />

    <xsd:element name="DisplayName" type="xsd:string" />

    <xsd:element minOccurs="1" name="DeviceKey" type="xsd:string" />

    <xsd:element name="Model" type="xsd:string" />

    <xsd:element name="Vendor" type="xsd:string" />

  </Configuration>

  <ModuleImplementation Isolation="Any">

    <Composite>

      <MemberModules>

        <DataSource ID="Seed" TypeID="QND.NetLibrary.MatchOID.DS">

          <IntervalSeconds>$Config/IntervalSeconds$</IntervalSeconds>

          <SyncTime>$Config/SyncTime$</SyncTime>

          <Port>$Config/Port$</Port>

          <IP>$Config/IP$</IP>

          <CommunityString>$Config/CommunityString$</CommunityString>

          <Version>$Config/Version$</Version>

          <ObjectOID>.1.3.6.1.4.1.41091.1.1</ObjectOID>

        </DataSource>

        <ConditionDetection ID="MapToDiscovery" TypeID="System!System.Discovery.ClassSnapshotDataMapper">

          <ClassId>$MPElement[Name='QND.MyNetwork.Device']$</ClassId>

          <InstanceSettings>

            <Settings>

              <Setting>

                <Name>$MPElement[Name='System!System.Entity']/DisplayName$</Name>

                <Value>$Config/DisplayName$</Value>

              </Setting>

              <Setting>

                <Name>$MPElement[Name='Network!System.NetworkManagement.Node']/DeviceKey$</Name>

                <Value>$Config/DeviceKey$</Value>

              </Setting>

              <Setting>

                <Name>$MPElement[Name='Network!System.NetworkManagement.Node']/Model$</Name>

                <Value>$Config/Model$</Value>

              </Setting>

              <Setting>

                <Name>$MPElement[Name='Network!System.NetworkManagement.Node']/Vendor$</Name>

                <Value>$Config/Vendor$</Value>

              </Setting>

            </Settings>

          </InstanceSettings>

        </ConditionDetection>

      </MemberModules>

      <Composition>

        <Node ID="MapToDiscovery">

          <Node ID="Seed" />

        </Node>

      </Composition>

    </Composite>

  </ModuleImplementation>

  <OutputType>System!System.Discovery.Data</OutputType>

</DataSourceModuleType>

3.       We must define our discovery rule targeted at the generic System.Network.Node class

  <Discovery ID="QND.MyNetwork.Device.Discovery" Enabled="true" ConfirmDelivery="false" Remotable="true" Priority="Normal" Target="Network!System.NetworkManagement.Node">

    <Category>Discovery</Category>

    <DiscoveryTypes>

      <DiscoveryClass TypeID="QND.MyNetwork.Device" />

    </DiscoveryTypes>

    <DataSource ID="DS" TypeID="QND.MyNetwork.Discovery.DS">

      <IntervalSeconds>43200</IntervalSeconds>

      <SyncTime />

      <Port>$Target/Property[Type="Network!System.NetworkManagement.Node"]/PortNumber$</Port>

      <IP>$Target/Property[Type="Network!System.NetworkManagement.Node"]/SNMPAddress$</IP>

      <CommunityString>$RunAs[Name="Network!System.NetworkManagement.Snmp.MonitoringAccount"]/CommunityString$$Target/Property[Type="Network!System.NetworkManagement.Node"]/VirtualCommunitySuffix$</CommunityString>

      <Version>$Target/Property[Type="Network!System.NetworkManagement.Node"]/SNMPVersion$</Version>

      <DisplayName>$Target/Property[Type="Network!System.NetworkManagement.Node"]/sysName$</DisplayName>

      <DeviceKey>$Target/Property[Type="Network!System.NetworkManagement.Node"]/DeviceKey$</DeviceKey>

      <Model>MyNetwork Device Model XYZ</Model>

      <Vendor>QND</Vendor>

    </DataSource>

  </Discovery>

This is it. But let’s review the key points here to make this process really generic. The QND.NetLibrary.MatchOID.DS is a composition of a standard scheduler, with the generic SNMP probe configured to SNMP GET (<Walk>false</Walk>) the RFC standard sysObjectID, the result is filtered to return just the devices with a matching OID ($Config/ObjectOID$). It can be underlined the isolation of SNMPv3 parameters, since it’s highly improbable we will use different credentials when running our workflows from the credential used to discover the device. This will make our subsequent workflows easier to read. This is a generic module that, if this series will continue, will add to a generic Network Library management pack.

The actual discovery datasource builds on the QND.NetLibrary.MatchOID.DS sets the OID to be matched and maps the resulting SNMP data to the QND.MyNetwork.Device class. As you can read this is where we can add some more data to the discovered properties (for example the model information). The mapping is done using a standard System.Discovery.ClassSnapshotDataMapper.

Finally the actual discovery rule, this rule will hit every single network device discovered so far so I would suggest to schedule it once or twice a day, for every device it will perform a SNMP GET and check if the returned OID is the MyNetwork.Device one, if so the device is discovered.

From here we can go on building our specific monitor, but this is the topic for another post.

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

To my knowledge at the time of this writing the differences between generic and certified devices are:

generics don’t have vendor and model (it can be easily added, most vendor publish the device model in SNMP)

generics don’t have CPU and memory monitoring, thus you don’t have the device in the top 10 cpu usage devices in the generic dashboard. (If the device exposes the CPU counter it can be easily added.)

generics don’t have the discovery of sensors and fans. Currently this is not a significant difference since they’re not monitored in any case. Adding this can be challenging depending on how they’re exposed in SNMP. The snmp modules in OpsMgr are not engineered for discovery scenarios and heterogeneous table management can be tricky, but I will present a couple of solutions for it.

generics don’t have trap based discovery, this can be a curse rather than a benefit. (see http://nocentdocent.wordpress.com/2013/03/26/network-discovery-reference-sysctr-scom/)

From the previous list you can correctly conclude that discovery works for all devices (certified and generic). If the device implements RFC standards for SNMP port monitoring then all the ports are discovered and when applicable connected to other managed devices (network or computer), this means all the tricky work needed for creating the network topology and understanding the links between devices is done regardless of certified or generic status for the device.

Returning to the original question “how can I certify my own generic devices?”, the answer is: no you cannot certify your own generic devices, but you can bring them on par with certified ones if they expose the management capabilities via SNMP.

To add some more details, which devices are certified and which are generic is defined in the network discovery module, this happens to be code Microsoft licensed from EMC under the SMARTS agreement, the configuration of this code is not documented and it is not supported to bring in any mod. This means your devices will remain GENERIC in console even if they can be better monitored than certified ones.

Actually a device can be forced to CERTIFIED via a custom discovery rule, but this property will be overwritten every time the network discovery runs and, in my opinion, will just add confusion. Certified or Generic is the OpsMgr certification level so leave it alone and let’s focus on monitoring.

This is the first in a serie of articles on SNMP device monitoring, I don’t know how many of them yet, and I don’t know where the code will be posted, there are several possibilities here: I can use technet gallery, I can open a codeplex project or I can just publish the snippets without the full sourcecode. I can even consider, with some help, to merge the xSNMP modules and integrate them for OpsMgr 2012. It really depends on the community feedback I will get.

This first chapter will set the basics, the references and the table of contents for future chapters.

Table of contents

Chapter 1 – Creating SNMP monitoring Management Packs for System Center 2012 Operations Manager (this article)

Chapter 2 – Discovery: identifying the device.

Chapter 3 – Memory and processor monitoring. Bringing the generic device almost on par.

Chapter 3 – Device components discovery.

Chapter 4 – Device components advanced discovery and extending the OpsMgr SNMP module with powershell.

The basics

The basic process for adding a new device is all around knowingly how the device responds to SNMP queries; this is typically documented in vendor published or standard MIB files. Without this first step there’s nothing that can be done. Once we get there it’s pretty straightforward to use the SNMP modules of Operations Manager 2012 to broaden the manageability of the devices. This typically implies performing a targeted discovery to uniquely identify the device, this is normally achieved via the sysObjectID of the device. Once we can uniquely identify the device we can target all the further monitors and rules to our specific device type.

In summary:

1.       Get the device SNMP tree

2.       Get a unique identifier for the device, this typically is the sysObjectID

3.       Define a new class for the device inherited from System.Network.node

4.       Write a discovery rule to identify the device (remember the first discovery is performed in any case by the OpsMgr standard modules)

5.       Target any further discovery, rule or monitor to the device class

The tools

To obtain the full device SNMP tree you need tools to perform queries and have a graphical representation of MIB files. The following is my short list:

MIB Browser: Manage Engine http://www.manageengine.com/free-tools.html

SNMP Query Utilities and SNMP extensions: http://sharpsnmplib.codeplex.com/

SNMP Simulator: http://snmpsim.sourceforge.net/

Online MIB definitions: http://support.ipmonitor.com/mibs_byoidtree.aspx, http://tools.cisco.com/Support/SNMP/do/BrowseOID.do

The OpsMgr module types and classes

The key management packs for network monitoring are:

·         System.SNMP.Library – defines basic SNMP types and data sources

·         System.NetworkManagement.Library – defines all the classes for certified devices and advanced data sources

·         System.NetworkManagement.Monitoring – defines actual monitoring for certified devices

You can find the documentation for relevant modules on MSDN (see references) but the above management pack are really useful to understand how network monitoring works and to get samples to build your own code.

Since we’re going to write our own management packs for specific devices, it’s fundamental to remember which are the base classes that we need to inherit from. For network management we have two base classes:

·         System.NetworkManagement.Node – this is the actual device, our device class will inherit from Node

·         System.NetworkManagement.LogicalDevice – this is a generic device component (such as processor, or fan) we can use this to model our device, in several cases we have more specialized classes such as System.NetworkManagement.Processor or System.NetworkManagement.Memory that can be used for specific and standard components

While the base properties of the Node class are already discovered for us, remember the first discovery is up to OpsMgr standard modules (we’re just going to implement a specialized discovery), the LogicalDevice class has a couple of key properties:

-          Key – it must be unique within the hosting device, the network monitor standard is <abbreviation>-<device address>/<index> (MEM-172.31.1.1/1). When possibile we will instead use the component OID so it can be direclty used in rules and monitors.

-          Index – it’s the device Index in the SNMP tree, useful when there is more than one component of that type per device to build the full SNMP OID. We’ll see plenty of such examples in code.

<ClassType ID=“System.NetworkManagement.LogicalDevice“ Accessibility=“Public“ Abstract=“true“ Base=“System!System.LogicalHardware“ Hosted=“false“ Singleton=“false“>

          <Property ID=“Description“ Type=“string“ Key=“false“ CaseSensitive=“true“ Length=“256“ MinLength=“0“ />

          <Property ID=“Index“ Type=“string“ Key=“false“ CaseSensitive=“true“ Length=“256“ MinLength=“0“ />

          <Property ID=“Key“ Type=“string“ Key=“true“ CaseSensitive=“true“ Length=“256“ MinLength=“0“ />

Resources and references

MIB Catalog: http://www.ndt-inc.com/SNMP/MIBsByRFC.html

Migrating Operations Manager 2007 R2 Network Monitoring

SNMP Event Wizard in OpsMgr on http://technet.microsoft.com/en-us/library/hh563870.aspx

SNMP Modules reference on MSDN http://msdn.microsoft.com/en-us/library/jj130169.aspx

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

I know the product group is aware of this issue and is working to release an updated package for gateways, in the mean time you can get rid of this issue copying the agent management directory from one of the upgraded Management Server to the corresponding directory on gateways.

Just to be clearer this is an example for the x64 (amd64) agent binaries, on the management server check the %programfiles%\System Center 2012\Operations Manager\Server\AgentManagement\amd64 folder for updated files (dated March 22nd)

What you need to do is copy these files to the corresponding directory on all your gateways. This needs to be done for the agent platforms.

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

Fundamentals

There are a few rules you need to keep in mind for network discovery:

·         Network discovery must be run from a management server (gateways are management servers)

·         Just one network discovery rule per management server is allowed.

·         Network discovery can be recursive or explicit. Recursive network discovery takes one or more seed devices and tries to find connected devices from the ARP tables of the seeds; in explicit discovery, every device must be defined in the discovery rule. Recursive discovery doesn’t work for SNMPv3 devices.

·         Network discovery can use SNMP, ICMP or both. If you want to have the device response time dashboard widget loaded with data, ICMP must be included in discovery.

·         Network discovery, by default, can be triggered by changes in the monitored devices. This can be a bless or a curse, you just need to know this happens. More info in Notes from the field.

·         To make it possible for OpsMgr to build the relationships between the network devices and the connected servers the OS management packs must be imported and NIC discovery must be enabled (it is by default). AFAIK it correlates the MAC address of the servers NICs to the MAC address found on the device port. (didn’t find any explicit documentation on this yet)

·         Device are named in OpsMgr using DNS resolution on the following items in the order listed, the first one to succeed wins 1) Loopback IP 2) sysName 3) Public IP 4) Private IP 5)SNMP Agent IP. You can force the use of sysName modifying the discovery configuration file, not sure if it’s supported (http://blogs.inframon.com/post/2012/07/15/How-to-use-the-MIB2-System-Name-for-a-device-in-SCOM-2012.aspx)

·         Network discovery rules are persisted in an unsigned management pack Microsoft.SystemCenter.NetworkDiscovery.Internal

·         Operations Manager specific firewall rules must be enabled (see Troubleshooting)

·         We can basically have two discovery actions:

-          Full discovery, that is a manual or scheduled triggered discovery

-          Limited discovery, that is a single device or limited devices discovery triggered by a manual single device rediscover task or by an SNMP trap.

Troubleshooting

The first level of information gathering for troubleshooting is done via the eventlog on the discovering management server, the whole process is well traced. You can also follow the discovery events from the OpsMgr console under Operations Manager\Network Discovery folder.

 I traced a normal network discovery session, the events are typically logged in pairs in the form of start/stop or open/close events:

A typical full discovery is composed by the following events:

The second level of troubleshooting involves turning on tracing, this can be done modifying the network discovery configuration file by default in C:\Program Files\System Center Operations Manager 2012\Server\NetworkMonitoring\conf\discovery\discovery.conf. The discovery.conf file is very well documented, I don’t know if any mod is supported if not asked by CSS, so it’s always a good idea to make a backup copy before any change and remember that any OpsMgr fix can overwrite the changes. What Stefan’s wrote (see references) has slightly changed in RTM and SP1, now the resulting log file is in c:\windows\temp and we have a couple of new switches. To quickly recap, to perform a full debugging session:

1. Remove all the files in the Program Files\System Center Operations Manager 2012\Server\ networkmonitoring\local\repos directory
2. Edit networkmonitoring\conf\discovery\discovery.conf

Case is significant.

3. Stop and restart the health service

4.       Turn on OM tracing

a.       go to the Program Files\System Center Operations Manager 2012\Server\Tools directory

b.      Run StartTracing.cmd DBG

5.       Run discovery

6.       Turn off OM tracing

a.       go to the Program Files\System Center Operations Manager 2012\Server\Tools directory

b.      Run StopTracing.cmd

7.       Turn off network discovery engine logging

a.       Edit networkmonitoring\conf\discovery\discovery.conf

Case is significant.

b.      Stop and restart the health service

In certain cases, more tracing is needed on the OpsMgr side, in this case, you have to call CSS and get a trace information file to add to the public ones. I’m not authorized to share such a file.

Notes from the field

The typical reason for discovery failure is a firewall issue on the management server, here the rule is pretty straightforward: the Operations Manager specific rules *must* be enabled, it doesn’t matter if other rules for the same protocols are enabled, the OpsMgr specific rules need to be turned on. Another important notice is: you can, if you really want, disable the firewall but you better not disable the firewall service. In the latter case I’ve seen all sort of strange behaviors.

The rules are easily identifiable

The second issue is more subtle. I anticipated that discoveries can be triggered by specific configuration change traps from the devices, this happens if:

-          The device is configured to send trap to the monitoring management server (discovery and monitoring management servers can be different, but this is another story)

-          The change traps are enabled

When this happens a limited network discovery is triggered (now you know you’ll get event id 12001). So far so good. This feature has two drawbacks:

-          If the device(s) sends too many traps (the discovery is triggered once for every trap) your management severs is backlogged of limited discovery tasks that in turns have an impact on CPU and memory usage and block scheduled full discovery. This is not a situation you want to be in.

-          There’s a bug (still there in SP1 UR1) that when a limited discovery is triggered on a device the computer connections associated to other devices are lost (event id 12199), so your topology is basically gone.

Personally, I prefer not to use trap based discovery, to achieve this you can:

-          Filter out traps with the windows firewall

-          Not configure devices to send traps to management servers, currently traps are not used for network monitoring (this can change in future and you can have a custom MP that uses traps)

-          Disable the discovery rules that, btw, are not marked as discovery rules but rather are standard ones (sigh)

Currently I identified the following rules:

Trap Received (3Com Card Inserted) – System.NetworkManagement.3Com.Node.CardInsertedEvent

Trap Received (3Com Module Inserted) – System.NetworkManagement.3Com.Node.ModuleInsertedEvent

Trap Received (Cisco Configuration Change) –

System.NetworkManagement.Cisco.Node.ConfigurationChanged

Trap Received (Cisco Configuration Management Event)-

System.NetworkManagement.Cisco.Node.ConfigurationManagementEvent

Trap Received (Cisco FRU Inserted) – System.NetworkManagement.Cisco.Node.FRUInsertedEvent

Trap Received (Cisco FRU Removed) – System.NetworkManagement.Cisco.Node.FRURemovedEvent

Trap Received (Cisco Module Status Change) – System.NetworkManagement.Cisco.Node.ModuleStatusChange

Trap Received (Cisco Reload) – System.NetworkManagement.Cisco.Node.Reload

A sample rule is structured as the following:

      <Rule ID=“System.NetworkManagement.Cisco.Node.ConfigurationManagementEvent“ Enabled=“true“ Target=“NetworkLibrary!System.NetworkManagement.Cisco_Node“ ConfirmDelivery=“false“ Remotable=“true“ Priority=“Normal“ DiscardLevel=“100“>

        <Category>Discovery</Category>

        <DataSources>

          <DataSource ID=“Trap“ TypeID=“NetworkLibrary!System.NetworkManagement.TrapTriggerProvider“>

            <IP>$Target/Property[Type="NetworkLibrary!System.NetworkManagement.Node"]/SNMPAddress$</IP>

            <TriggerOID>.1.3.6.1.4.1.9.9.43.2.0.1</TriggerOID>

          </DataSource>

        </DataSources>

        <WriteActions>

          <WriteAction ID=“WA“ TypeID=“NetworkLibrary!System.NetworkManagement.TrapDiscoveryRequestPublishData“ />

        </WriteActions>

      </Rule>

References

My fellow MVP Stefan wrote a long ago on network discovery troubleshooting: http://www.code4ward.net/main/Blog/tabid/70/EntryId/105/Troubleshooting-Network-Discovery-in-SCOM-2012.aspx

Inframon tip for using sysName instead of DNS name http://blogs.inframon.com/post/2012/07/15/How-to-use-the-MIB2-System-Name-for-a-device-in-SCOM-2012.aspx

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

Currently I’m not working with hosters, so these notes will only incidentally take into account network isolation. In any case one thing is sure network virtualization is badly needed for a private cloud deployment, this technology is not only for hosters. In fact thanks to network virtualization it is possible to manage the network connection in a standardized way and optimize physical network utilization.

The key terms for Network Virtualization in VMM in the order they should be defined are:

Logical Network – abstraction of the underlying network, logical networks are “networks roles” such as “production”, “dmz”, “management”, …. A VM is connected to a Logical Network via a VM Network reference and a Logical Switch

Network site – simply a network site. A Logical Network is typically implemented in multiple sites, so Logical networks contain multiple site, each site has its own associated subnets (and VLAN). Network sites are typically scoped to specific host groups.

IP Address pool – a range of IP addresses associated to a network site. Similar thing for MAC address pool. At provisioning time a VM can allocate an IP address from the pool the VM is connected to.

Native Uplink port profile – defines the characteristics of the physical connection between a logical switch and the physical network. It’s an “uplink” in networking terms. It defines: teaming behavior if any, which logical network are connected by means of the port.

Native Virtual Network Adapter Port Profiles – defines the virtual network adapter behavior (offloading, security, bandwidth). The profiles applies both to VM virtual NICs *and* host virtual NICs.

Port Classification – is a friendly name for the Virtual Port, it is commonly associated to a Native Virtual Network Adapter Port Profile during the definition of Virtual Port in Logical Switches.

Logical switch – defines a switch (it’s the *new* virtual switch). It must be considered like a network switch, it defines Uplink ports and Virtual Ports. It defines if teaming is supported or not. It can support extensions such as filtering or captures. Uplink ports are the connections towards the physical network (via a Network site and as a consequence a Logical Network). Uplink ports are defined through Native Uplink Port Profiles; Virtual Ports are defined through Port Classification and Native Virtual Network Adapter Port profiles.

Virtual machine Network – is the abstraction for the network the VMs connect to. It can be a standard network or an isolated network, in both cases it refers to a Logical Network. Isolated networks are commonly used in hosting scenarios when multiple IP subnets in the same IP address space need to coexist in an isolated manner.

Let’s start with a relationship diagram between all the physical and logical entities involved in network virtualization with vmm.

I designed this reference schema to try to give a logical order to network virtualization, from the schema we can deduce a few clear rules:

-          Every VM can have one or more Virtual Network Adapter

-          Every VA is bound to a VM Network and to a logical switch with an optional port classification

o   The VM Network can be a standard or an isolated one, in the latter case an isolated subnet needs to be chosen

o   The Logical Switch sets which network sites are reachable through the uplink port profile

o   The port classification sets the properties of the Virtual Netwok Adapter and it’s optional

-          Every VM Network is bound to a Logical Network

-          Every Logical Network can have multiple sites who in turns can have multiple subnets and IP Address Pool. Sites can be bound to specific host groups.

-          Logical switches need to be defined at the host level, they set which physical NICs are used and which Network Sites and Logical Networks (as a consequence) are reachable from the Logical Switch. More NICs can be teamed is the Logical Switch is configured to support teaming and the NICs are bound to the same uplink profile.

-          The use network convergence i.e. be able to use a network channel for multiple distinct purposes Virtual Network Adapter need to be added to the Logical Switch on the Host, every Virtual Network Adapter is shown as virtual NIC at the host level. Since every Virtual Network Adapter can have its own classification, network traffic and bandwidth can be different for each Virtual Network Adapter. Important: Virtual Network Adapter created for the Logical Switches at the host level are used for exposing to the host different networks, these adapters are not used by VMs.

A VM can be connected only to a Virtual Machine Network for which a Logical Switch with an uplink port to a Network Site contained in the Logical Network referenced by the Virtual Machine Network has been defined on the host. OK just follow the green arrows

Lessons learned and FAQ

Every physical NIC can only be associated to one logical switch. 1 NIC : 1 Logical Switch

Every Logical switch can use multiple NICs (teamed or not): 1 Logical Switch : n NICs

As a best practice define at least one logical switch for each different physical connection needed, for example if I want a host and the hosted VMs to communicate using three different NICs without using teaming (production, management, dmz) I would define three different logical switches. The logical switch definition is determined by the physical NICs configuration on the hosts, also.

Once a Logical switch is used by a VM or a Host the Logical switch properties cannot be changed.

The IP address pools associated to Network Sites are used only when a new VM is provisioned starting from a VM Template or a Service Template.

Currently it is not possible to have overlapping IP address range between the provider and the customer.

Currently there’s no way to hide from VMs a logical switch used for converged networks at the host level.

Network teams must be configured and used via VMM and not via the OS.

Migrating from legacy virtual switches to logical switches and network virtualization

While for green fields deployment it’s only a matter of proper design, in case of an existing virtualization infrastructure things are a little more complicated. I’m assuming an Hyperv 2008 R2 to Hyperv 2012 migration.

I didn’t find a way to migrate from the legacy model (virtual switch based) to the network virtualization model (logical switch based) without disrupting service operations: at the very least VMs need to be stopped before the virtual network adapter can be reconfigured to use a logical switch.

Migration models:

-          Side by side or leap frog – using a spare brand new Windows Server 2012 server

-          In place using the existing hosts

The first model is very similar to a green field deployment, once the virtualized network infrastructure has been defined in VMM a WS2012 host is prepared and configured. The VMs are migrated to the new host using Storage migration (typically with a downtime of a few dozen seconds). Once the first legacy host is evacuated, it can be upgraded and integrated in network virtualization. VMs can then be moved back without downtime using shared nothing Live Migration. The process is repeated for every host to be upgraded. The same process can be applied to failover clusters.

The in place model requires downtime, if there’s spare capacity in the data center it can be reported to the side by side model. If this is not an option, the risk goes up and the procedure is basically the following:

-          Upgrade the host to WS2012 (inplace upgrade is supported). Downtime.

-          Get a free NIC, for example removing a NIC from an existing team. Build the first Logical Switch.

-          Using powershell rebind the VMs to the new switch. At this point the NIC(s) used by the legacy switch can be recycled. Downtime, to rebind the virtual network adapter from the virtual switch to the logical switch the VMs need to be turned off.

-          Repeat the process for every existing Virtual Switch.

If you want to migrate an existing dedicated management NIC to a converged network model you cannot use a teaming Logical Switch and migrate the existing NIC configuration. When the team is created by VMM the DNS configuration is lost, the new network is identified as public and basically any communication with host is lost, thus leaving the host only partially configured.

Network Isolation

tbd

References

http://blogs.technet.com/b/scvmm/archive/2013/01/08/virtual-networking-in-vmm-2012-sp1.aspx

http://blogs.technet.com/b/scvmm/archive/2013/02/14/networking-in-vmm-2012-sp1-logical-networks-part-i.aspx

Anyone who tried the dashboards outside a lab or poc environment experimented with the sluggishness of the performance related views (widget). While attending the MVP Summit (virtually) I took a couple of hours off to drill through the issue. My aim was to give a task based dashboard with the basic performance indicators:

The dashboard included:

-          Cpu usage

-          Memory usage

-          The response time for all disks

-          The bandwidth usage for all NICs

Alas, the dashboard was so slow to be unusable and sometimes some counters weren’t displayed at all.

To begin my troubleshooting effort I started from the MP definition (Microsoft.SystemCenter.Visualization.Library) to rapidly get to the SQL Server side: in fact all the data interaction is performed with the Data warehouse using newly defined stored procedures with the SDK schema/prefix. Time for a good SQL Profiler session, I had all the information I needed to setup a proper trace. What I found has been astonishing, tens even hundreds of calls like the following for every dashboard refresh:

exec SDK.Microsoft_SystemCenter_Visualization_Library_SinglePerformanceDataSeriesGet @ManagementGroup=’6934C4FC-5C84-2C84-C5A0-88726754720D’,@StartTime=’2013-02-17 14:57:26.730′,@EndTime=’2013-02-20 14:57:26.730′,@ManagedEntityGuid=‘CD4CBEB2-85A0-0F82-DFC3-12186EDB7F5A’,@PerfRuleInstanceRowId=132633,@NumberOfDataPoints=100,@RequestedDataPointType=4

It was clear something was wrong (badly wrong), not only the perf widget is slow it taxes the SQL engine as well. At the beginning of the trace another stored procedure was called, this time 4 times with different parameters: clearly once for every widget in the dashboard.

exec SDK.Microsoft_SystemCenter_Visualization_Library_PerformanceCounterListByManagedEntityUsingContainerME @ManagementGroup=’6934C4FC-5C84-2C84-C5A0-88726754720D’,@ContainerManagedEntityXml=N’<ManagedEntityIds>

  <ManagedEntityId>cd4cbeb2-85a0-0f82-dfc3-12186edb7f5a</ManagedEntityId>

</ManagedEntityIds>’,@ObjectNamePattern=N’Network Adapter’,@CounterNamePattern=N’PercentBandwidthUsedTotal’,@InstanceNamePattern=N’%’

Trying to execute each one of these showed the culprit:

-          The memory one returned just one row in no time

-          The CPU two rows

-          The network about 200 rows in 30”

-          The disc about 5100 rows in more than 3’. The dashboard are supposed to refresh every 60”, ahem.

What was happening is that the stored procedure returned one row for every instance (and rule, more on this later) of the requested performance counter. The dashboard logic in turn, for every single row, executed the SinglePerformanceDataSeriesGet, which retuned data only for the instances effectively present on the computer targeted by the dashboard (to be precise present on the targeted managed entity). This is what I call a bad (very bad) design.

Time for some serious TSQL code review for SDK.Microsoft_SystemCenter_Visualization_Library_PerformanceCounterListByManagedEntityUsingContainerME.

The results of the review are embarrassing, in summary:

-          Management pack versioning was not considered so multiple performance rule instances are returned for the same counter. This happens when the MP defining the rule has been upgraded one or more time, quite a common scenario in a production environment.

-          Managed entity life time was not considered, so dismissed managed entity were taken into account, contributing to execution time.

-          All the performance rule instances were retuned even if such instance doesn’t exist on the target managed entity

-          Lastly the logic never consider the chance to have multiple collecting rules for the same performance counter

Not bad for a single stored procedure.

Before going further, I want to make clear the scope of post: I just modified the stored procedure needed for my specific goal, there are other similar stored procedures that, I guess, suffer of the same issues. I’m going to explain how I modified the stored procedure, if you want to go this way you just remember it is at your own risk.

Lastly I won’t post the complete TSQL, you must go through the process of modifying the stored procedure by yourself. Remember any update to the MP will overwrite the stored procedure.

Step 1. Create a script for ALTER of the stored procedure

Step 2. In SQL management studio rename the original stored procedure so that you can restore it if needed.

Step 3. Ready to modify the script the following are the required mod, they are all tagged with [QND] with a short description.

The first correction is to the statement that populates all the possible targets for the given performance counter, I added the check on validity and removed the explosion for all the instances

The second attack point is at the final select where we need to perform several task:

-          Filter only valid instances for the targets

-          Remove duplicated collection rules

-          Consider only valid objects just in case something has skipped from previous queries

The net result is perf widgets dashboard have acceptable performances, now. For example the selection of disk instances who returned more than 5K rows in more than 3’ now for a specific server return 4 rows (one for each disk) in 5”, in turn this generates 4 stored procedure calls instead of the 5K of the previous implementation.

With such a design is no mystery (now) what performance dashboards are so slow.

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

Log Name:      Operations Manager

Source:        HealthService

Date:          2/6/2013 9:22:42 AM

Event ID:      4509

Task Category: Health Service

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      XXXXXXXXXXXXXXXXXX

Description:

The constructor for the managed module type “Microsoft.SystemCenter.Cloud.OutsideInUnitModule.VSWebTestCondition” threw an exception. This module was running in rule “Microsoft.SystemCenter.Omonline.OutsideIn.VSWebTest.Performance.CollectTransactionResponseTime” running for instance “UGISC [US : CA-Los Angeles]” with id:”{F88050C8-B7C6-B36E-1309-5E344BCFB1BA}” in management group “XXXXXXX”.

The exception text is:

Microsoft.EnterpriseManagement.HealthService.ModuleException: Invalid Xml Config received. —> System.Xml.XmlException: ‘None’ is an invalid XmlNodeType. Line 1, position 284.

   at System.Xml.XmlReader.ReadStartElement(String name)

   at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.VSWebTestCondition.ReadConfiguration(XmlReader configurationReader)

   — End of inner exception stack trace —

   at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.VSWebTestCondition.ReadConfiguration(XmlReader configurationReader)

   at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.OutsideInTestResultCondition`1..ctor(ModuleHost`1 moduleHost, XmlReader configurationReader, Byte[] previousState)

   at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.VSWebTestCondition..ctor(ModuleHost`1 moduleHost, XmlReader configurationReader, Byte[] previousState)

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

When upgrading your ACS infrastructure to SP1 you’re likely to hit the following bug (actually there are two bugs). If this is the case you either have the ACS collector crashing every night when a new partition is created or you don’t have the views for Windows 2012 claim authentication.

A quick background: SP1 should bring your ACS database schema version to version 8, you can check the schema version in table dtConfig with Id 2:

In all my ACS upgrades it remains to schema version 7, this means the sql script DbUpgV7toV8.sql it has not been applied during the update. If have 100% repro on this, so I’m expecting your DB schema will still be at level 7.

Upgrade to level 8 introduces, among other minor mods, the new tables for claim authentication:

• dtClaimString
• dtOldResourceAttribute
• dtNewResourceAttribute
• dtUserClaim
• dtDeviceClaim

and the corresponding views (dv*) one for each partition present in the ACS database.

SP1 should as well upgrade some sql script, among them the DbCreatePartition.sql. The dbCreatePartition script creates a new set of tables and views for the new partition and updates the summary views where all the partitions are “unioned” together (i.e. [AdtServer].dv*).

The DBCreatePartion.sql sometimes doesn’t get updated, there’s no fixed rule here and I didn’t find an exact repro, I suspect this happens if the original sql has been modified, for example to include all the partitions (see http://support.microsoft.com/kb/2663919).

After the update completes we can find ourselves in two situations:

1. the schema has not been updated and the dbCreatePartion has
2. the schema and the dbCreatePartion has not been updated

In case 1. The ACS collector will stop when it creates a new partition, when it executes the dbCreatePartion and tries to create the summary view for claim authentication it ends in error since the claim authentication views for old partitions are missing.

You can find events similar to the following in the Operations Manager event log

Log Name:      Operations Manager

Source:        AdtServer

Date:          2/12/2013 02:01:31

Event ID:      4618

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:     

Description:

Error occured on database connection:

Status: 0x0409C002

ODBC Error:       208

ODBC State:      42S02

Message:           [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name ‘dvUserClaims_81971a6f_d0dc_477c_9316_81def5fe3ac2′.

Database:           SqlWriter

Connection:      Maintenance

Statement

Log Name:      Operations Manager

Source:        AdtServer

Date:          2/12/2013 02:01:31

Event ID:      4680

Task Category: None

Level:         Error

Keywords:      Classic

User:          NETWORK SERVICE

Computer:     

Description:

The Operations Manager 2007 audit collector service encountered a fatal error while attempting to perform a database operation. The Operations Manager Audit Collector service is unable to continue operation and is stopping.

Please check the log for previous events describing the nature of the problem.

To restore the collector to operation, you must troubleshoot Microsoft SQL Server or the Microsoft SQL data access components to find the cause of the error and resolve it, then restart the collector service

To solve this situation you can just restart the collector and it will work ok until next partition creation. Once all the old partitions are purged, it depends on your retention policy, the collector will no longer stop. Obviously this is not an ideal situation to be in, my *totally unsupported*  workaround is to manually apply the DbUpgV7toV8.sql to your ACS database, this will fix the issue once for all. (as usual do it at your own risk, the supported way is to open a CSS call)

In case 2. You don’t have any visible issue, simply the ACS database doesn’t have the views for claim based authentication. If you’re in case 2. You probably didn’t notice anything. To have the claim based authentication views in place you should return to case 1., or:

-          manually copy the correct DbCreatePartion.sql script

-          apply DbUpgV7toV8.sql to your ACS database

Remember: unsupported, do it at your own risk.

The SQL scripts I’m mentioning can be found in the ACS\AMD64 folder in the SP1 ISO/DVD and once installed in c:\windows\system32\sceurity\adtserver much like the following screenshot

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

Note. This is more a wiki page than a standard post, I will periodically keep it up to date on the subject.

Acknowledgment. I would like to thank Cameron Fuller for the thoroughful review of this post

Global Service Monitoring (GSM) is a welcome addition in System Center 2012 Service Pack 1 Operations Manager (OpsMgr). GSM allows Operations Manager to perform availability and performance tests for web applications using remote probes. These probes can be located both in your private cloud and in the public cloud. GSM is licensed within Software Assurance but currently only the trial subscription is available. It is expected that around April 2013 final subscriptions can be enrolled.

A good introductory article on the subject with a complete guide on how to setup the GSM environment can be found here: System Center Global Service Monitor: Getting Started. I just want to underline it’s a good idea to create a specific Resource Pool for GSM, in fact the management server(s) used for GSM need to access internet resources and such as they need a specific configuration.

Once the GSM subscription is activated/registered within the OpsMgr console it’s possible to start adding tests. This is done in the Operations Console in the authoring pane space through a two wizard driven management pack templates.

There are two types of test that can be used:

-          Single URL tests, basically a HTTP GET to a given URL

-          Multiple steps tests or Visual Studio Web Tests, where a complete interaction with the tested web site can be played repeatedly.

For the first tests external and internal probes can be defined, obviously to use external (from the public cloud) probes the web sites need to be internet facing. Internal probes can be defined as long as the probe system has an OpsMgr agent installed, for internal probes geo location can be defined so that they can be plotted on a world map much like external ones. All this process is well explained in a couple of posts so I won’t repeat here the step by step actions, rather I address the reader to the original posts:

-          Map Dashboard in SC 2012 Operations Manager

-          Using the Web Application Availability Monitoring to monitor Web Applications health

For the multiple step web tests use the Visual Studio web tests. Please note that only external probes can be defined, but these tests are more powerful than the web application availability monitors  and give much more control on how tests are performed. Since the Web Performance Test Engine works at the HTTP layer, it does not run client-side scripting like JavaScript or ActiveX controls.

Let’s first set the basics for Visual Studio Web Performance Test (WPT). WPT are available in Visual Studio 2012 Ultimate so, first thing to remember, you need Visual Studio Ultimate 2012 (or 2010) to build your Web Test. A 90 days trial can be downloaded from MSDN.

Visual studio has two major types of Web Tests:

-          Web Performance Test

-          Web Load Test

OpsMgr GSM can use Web Performance Test with the following limitations:

-          Thinking Time is not supported

-          Transactions are not shown in the result sets, but they are maintained if the result file is opened in Visual Studio.

-          Custom code tests are not supported. OpsMgr can only use WPT script files and not the built DLL. Hence you don’t need to “build” your WPT to use them with OpsMgr

-          Data binding is not supported, but parameters are

-          Reporting name is just ignored

WPT opens a wide range of possibilities regarding testing, in this post I will go through the basic ones, to have more info I would start from this tutorial on MSDN  Web Performance Test Walkthroughs.

More tuning

WPT results are files returned from the probes to the GSM management server, to be sure things work as expected it is a must to properly configure the Alert Attachments management pack (http://technet.microsoft.com/en-us/library/jj899889.aspx). To make it works properly you must remember to grant the Alert Attachments run as account change rights to the temporary directory of you default runas account on the Management servers selected. This is normally SYSTEM and the temporary directory for system is by default c:\windows\temp. This is not necessary if the runas account is local admin on the GSM management servers.

The tested web site

To show many of possibilities exposed by WPT and OpsMgr GSM I will setup a we test to perform a secure access to the web site of  User Group Italy – System Center www.ugisystemcenter.org. The UGI System Center web site is based on .net nuke community edition and indeed it’s pretty plain and simple, nevertheless it exposes enough functionality to be used as a test case.

Creating a Web Performance Test in Visual Studio

The process to create a proper web tests sequence is pretty simple and can be summarized as following:

·         Create a new visual studio project of type “Web Performance and Load Test Project”

·         In solution Explorer add a new Web Performance test

·         As soon as the new item is added a web browser window opens ready to track the navigation sequence

·         Just navigate as a normal user would be, trying to test all the aspects of the web site that interesting on a monitoring point of view

Basically that’s it, the web tests sequence is ready to be customized with thresholds and parameters.

Let’s start, with the new Visual Studio project

 

Then add a Web performance test to the project

And start browsing

Once you’re done the browsing sequence is ready to be inspected

Once the sequence is ready there at least a couple of useful settings to be put in place, firstly for every single step a series of validation rules can be set, the most basic of them is the response time. For every request, the following properties can be set (I crossed in red the ones that today are not supported)

The first property you want to set on many if not all requests is Response Time Goal, if left to the default (0) this goal isn’t checked.

When you set a response time goal you can set on a global basis, the tolerance value is in percentage.

It is highly probable the web tests have parameters, in my case I have the user name and password that I want to be parameterized, this is useful anytime some properties need to be changed frequently for testing purposes. To add parameters (a pairing between a keyword and its value) just click in “Context Parameters” node and type in the parameter name and its value:

 

Then you can spend the parameters in any request, in my case I need to replace the actual username and password in post fields for the login request. To do this just click on the appropriate node and in the value for the proper parameter add the newly defined context parameter:

Once you’re done the task sequence can be tested from within Visual Studio using the run test button, the result will be displayed with all its properties inside Visual Studio

These are the very basic steps to prepare your own web test. Once you familiarize with the Visual Studio environment you’ll see many more options that can be tuned to set up a proper web test.

Creating a Visual Studio Web test in Operations Manager

Once the web tests are ready it’s time to move to OpsMgr admin console and setup GSM monitoring. The process is simple and wizard driven, in summary:

·         In the authoring space select the Add monitoring wizard and choose the “Visual Studio Web test Monitoring” template

·         Give your tests a name and select a custom management pack to save the monitoring rules in

·         In the “What to monitor” tab add a new row and select the .webtest file generated with Visual Studio

·         In the “Where to monitor from” select the external locations to be used for monitoring. For VS web test it’s currently not possible to select internal locations.

·         Finally in the “view and configure tests” tab set the frequency of the tests and the alerting preferences

Let’s import the webtest file generated for UGI SystemCenter. Fire the Add Monitoring Wizard and begin as shown below:

Give it a name and select a custom management pack

Select the webtest file from the Visual Studio project folder

If something goes wrong with validation a blocking message appears which details what the offending request is (remember GSM currently doesn’t support all WPT constructs)

If there are parameters defined you can now give them a proper value as shown below for the example which expects a UserName and Password.

The select where to monitor from for this sample I’m going to select Paris and Los Angeles

Finally it is possible to tune the test frequency and alerting behavior

This is the result for our sample

When the Create button is clicked several actions starts under the hood:

·         The rules are written in the selected management pack, specifically a new container class is created and a new discovery with the content of the webtest file is added.

·         When the discovery runs the test are packed and sent to ‘https://gsm-prod.systemcenter.microsoft.com/TestConfigurationService&#8217;

·         A poller workflow starts and check every 60” on gsm-prod.cloudapp.net if there are results pending. If so the webtest synthetic results are downloaded. The synthetic results do not include the single steps details, just the overall transaction status and response time

As soon as the discovery kicks in, the testing locations appear in console

The best view to follow the tests is the “Test result dashboard”, you must remember the tests result details are not automatically downloaded, actually the message is self-explanatory. This is where things can fail if you didn’t have properly configured the Alert Attachments management pack.

A properly configured monitor will show a situation similar to the following screenshot. If you get there then your web sites are being monitored from the Microsoft cloud!

If a check is unsuccessful an alert is generated in console accordingly to the alert policy set during the Visual Studio Web Test wizard. The alert reports the failing probe / location and in the context why it failed:

Every time an alert is generated the detailed webtest result file is downloaded, the downloaded file can be tracked in alert history, alas the result details are not shown in the detail dashboard unless the download task is manually fired.

Importing data back in Visual Studio

When web test results are downloaded via the specific console task what’s downloaded is actually the same result file Visual Studio would have created, so to have full access to all the details the result file can be opened in Visual Studio.  The file path is reported in the download task results pane to locate the file and open it with Visual Studio.

The net result is a fully compliant Visual Studio web test result file

Known issues

If, for any reason, the test is marked as failed (red) the Total Transaction time performance counter is not collected for that iteration. This implies that if the test always breaches the goals set, total transaction time is never collected. I consider this a bug and I have reported it to the Product group.

If transactions are used in web tests and a request inside a transaction fails you got the test location shown as unhealthy but the web test results green:

Troubleshooting GSM

The GSM flow back and forth between the probes can be tracked using the Operations Manager event log on the GSM management server. The event source is “Health Service Modules Ex”, under normal circumstances the following events should be logged:

-          10018 “Global Service Monitor Modules:  Successfully discovered VSWebTest container” where you can find the number of tests created

-          10000 “Global Service Monitor Modules:  Successfully uploaded web tests to ‘https://gsm-prod.systemcenter.microsoft.com/TestConfigurationService’”

Other important events:

-          10021 “Global Service Monitor Modules:  The test result file downloaded successfully” when a web test result file is downloaded from the cloud

The Download Web Test task needs read/write access to the temporary directory of the Default Action Account (normally LSA, thus c:\windows\temp) and to the Alert Attachments share. From what I could determine the web test result file is first downloaded in the temporary directory using the agent default action account and then moved to the alert attachments share via the alert attachments runas account.

Useful resources

How to Configure Integration with IntelliTrace Historical Profiling in System Center 2012 SP1

Map Dashboard in SC 2012 Operations Manager

Using the Web Application Availability Monitoring to monitor Web Applications health

System Center Global Service Monitor: Getting Started

- Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

]]>