Global Service Monitoring – moving service testing to the cloud


Note. This is more a wiki page than a standard post, I will periodically keep it up to date on the subject.

Acknowledgment. I would like to thank Cameron Fuller for the thoroughful review of this post

Global Service Monitoring (GSM) is a welcome addition in System Center 2012 Service Pack 1 Operations Manager (OpsMgr). GSM allows Operations Manager to perform availability and performance tests for web applications using remote probes. These probes can be located both in your private cloud and in the public cloud. GSM is licensed within Software Assurance but currently only the trial subscription is available. It is expected that around April 2013 final subscriptions can be enrolled.

A good introductory article on the subject with a complete guide on how to setup the GSM environment can be found here: System Center Global Service Monitor: Getting Started. I just want to underline it’s a good idea to create a specific Resource Pool for GSM, in fact the management server(s) used for GSM need to access internet resources and such as they need a specific configuration.

Once the GSM subscription is activated/registered within the OpsMgr console it’s possible to start adding tests. This is done in the Operations Console in the authoring pane space through a two wizard driven management pack templates.

There are two types of test that can be used:

          Single URL tests, basically a HTTP GET to a given URL

          Multiple steps tests or Visual Studio Web Tests, where a complete interaction with the tested web site can be played repeatedly.

For the first tests external and internal probes can be defined, obviously to use external (from the public cloud) probes the web sites need to be internet facing. Internal probes can be defined as long as the probe system has an OpsMgr agent installed, for internal probes geo location can be defined so that they can be plotted on a world map much like external ones. All this process is well explained in a couple of posts so I won’t repeat here the step by step actions, rather I address the reader to the original posts:

          Map Dashboard in SC 2012 Operations Manager

          Using the Web Application Availability Monitoring to monitor Web Applications health

For the multiple step web tests use the Visual Studio web tests. Please note that only external probes can be defined, but these tests are more powerful than the web application availability monitors  and give much more control on how tests are performed. Since the Web Performance Test Engine works at the HTTP layer, it does not run client-side scripting like JavaScript or ActiveX controls.

Let’s first set the basics for Visual Studio Web Performance Test (WPT). WPT are available in Visual Studio 2012 Ultimate so, first thing to remember, you need Visual Studio Ultimate 2012 (or 2010) to build your Web Test. A 90 days trial can be downloaded from MSDN.

Visual studio has two major types of Web Tests:

          Web Performance Test

          Web Load Test

OpsMgr GSM can use Web Performance Test with the following limitations:

          Thinking Time is not supported

          Transactions are not shown in the result sets, but they are maintained if the result file is opened in Visual Studio.

          Custom code tests are not supported. OpsMgr can only use WPT script files and not the built DLL. Hence you don’t need to “build” your WPT to use them with OpsMgr

          Data binding is not supported, but parameters are

          Reporting name is just ignored

WPT opens a wide range of possibilities regarding testing, in this post I will go through the basic ones, to have more info I would start from this tutorial on MSDN  Web Performance Test Walkthroughs.

More tuning

WPT results are files returned from the probes to the GSM management server, to be sure things work as expected it is a must to properly configure the Alert Attachments management pack (http://technet.microsoft.com/en-us/library/jj899889.aspx). To make it works properly you must remember to grant the Alert Attachments run as account change rights to the temporary directory of you default runas account on the Management servers selected. This is normally SYSTEM and the temporary directory for system is by default c:\windows\temp. This is not necessary if the runas account is local admin on the GSM management servers.

The tested web site

To show many of possibilities exposed by WPT and OpsMgr GSM I will setup a we test to perform a secure access to the web site of  User Group Italy – System Center www.ugisystemcenter.org. The UGI System Center web site is based on .net nuke community edition and indeed it’s pretty plain and simple, nevertheless it exposes enough functionality to be used as a test case.

Creating a Web Performance Test in Visual Studio

The process to create a proper web tests sequence is pretty simple and can be summarized as following:

·         Create a new visual studio project of type “Web Performance and Load Test Project”

·         In solution Explorer add a new Web Performance test

·         As soon as the new item is added a web browser window opens ready to track the navigation sequence

·         Just navigate as a normal user would be, trying to test all the aspects of the web site that interesting on a monitoring point of view

Basically that’s it, the web tests sequence is ready to be customized with thresholds and parameters.

Let’s start, with the new Visual Studio project

clip_image002[4]

 

Then add a Web performance test to the project

clip_image004[4]

And start browsing

clip_image006[4]

Once you’re done the browsing sequence is ready to be inspected

clip_image008[4]

Once the sequence is ready there at least a couple of useful settings to be put in place, firstly for every single step a series of validation rules can be set, the most basic of them is the response time. For every request, the following properties can be set (I crossed in red the ones that today are not supported)

clip_image010[4]

The first property you want to set on many if not all requests is Response Time Goal, if left to the default (0) this goal isn’t checked.

When you set a response time goal you can set on a global basis, the tolerance value is in percentage.

clip_image012[4]

It is highly probable the web tests have parameters, in my case I have the user name and password that I want to be parameterized, this is useful anytime some properties need to be changed frequently for testing purposes. To add parameters (a pairing between a keyword and its value) just click in “Context Parameters” node and type in the parameter name and its value:

clip_image013[4]

clip_image014[4] 

Then you can spend the parameters in any request, in my case I need to replace the actual username and password in post fields for the login request. To do this just click on the appropriate node and in the value for the proper parameter add the newly defined context parameter:

clip_image016[4]

clip_image018[4]

Once you’re done the task sequence can be tested from within Visual Studio using the run test button, the result will be displayed with all its properties inside Visual Studio

clip_image020[4]

These are the very basic steps to prepare your own web test. Once you familiarize with the Visual Studio environment you’ll see many more options that can be tuned to set up a proper web test.

clip_image022[4]

Creating a Visual Studio Web test in Operations Manager

Once the web tests are ready it’s time to move to OpsMgr admin console and setup GSM monitoring. The process is simple and wizard driven, in summary:

·         In the authoring space select the Add monitoring wizard and choose the “Visual Studio Web test Monitoring” template

·         Give your tests a name and select a custom management pack to save the monitoring rules in

·         In the “What to monitor” tab add a new row and select the .webtest file generated with Visual Studio

·         In the “Where to monitor from” select the external locations to be used for monitoring. For VS web test it’s currently not possible to select internal locations.

·         Finally in the “view and configure tests” tab set the frequency of the tests and the alerting preferences

Let’s import the webtest file generated for UGI SystemCenter. Fire the Add Monitoring Wizard and begin as shown below:

clip_image024[4]

Give it a name and select a custom management pack

clip_image026[4]

Select the webtest file from the Visual Studio project folder

clip_image028[4]

clip_image030[4]

If something goes wrong with validation a blocking message appears which details what the offending request is (remember GSM currently doesn’t support all WPT constructs)

clip_image032[4]

If there are parameters defined you can now give them a proper value as shown below for the example which expects a UserName and Password.

clip_image034[4]

The select where to monitor from for this sample I’m going to select Paris and Los Angeles

clip_image036[4]

Finally it is possible to tune the test frequency and alerting behavior

clip_image038[4]

This is the result for our sample

clip_image040[4]

When the Create button is clicked several actions starts under the hood:

·         The rules are written in the selected management pack, specifically a new container class is created and a new discovery with the content of the webtest file is added.

·         When the discovery runs the test are packed and sent to ‘https://gsm-prod.systemcenter.microsoft.com/TestConfigurationService’

·         A poller workflow starts and check every 60” on gsm-prod.cloudapp.net if there are results pending. If so the webtest synthetic results are downloaded. The synthetic results do not include the single steps details, just the overall transaction status and response time

As soon as the discovery kicks in, the testing locations appear in console

clip_image042[4]

The best view to follow the tests is the “Test result dashboard”, you must remember the tests result details are not automatically downloaded, actually the message is self-explanatory. This is where things can fail if you didn’t have properly configured the Alert Attachments management pack.

clip_image044[4]

A properly configured monitor will show a situation similar to the following screenshot. If you get there then your web sites are being monitored from the Microsoft cloud!

clip_image046[4]

If a check is unsuccessful an alert is generated in console accordingly to the alert policy set during the Visual Studio Web Test wizard. The alert reports the failing probe / location and in the context why it failed:

clip_image048[4]

Every time an alert is generated the detailed webtest result file is downloaded, the downloaded file can be tracked in alert history, alas the result details are not shown in the detail dashboard unless the download task is manually fired.

clip_image050[4]

Importing data back in Visual Studio

When web test results are downloaded via the specific console task what’s downloaded is actually the same result file Visual Studio would have created, so to have full access to all the details the result file can be opened in Visual Studio.  The file path is reported in the download task results pane to locate the file and open it with Visual Studio.

clip_image052[4]

The net result is a fully compliant Visual Studio web test result file

clip_image054[4]

Known issues

If, for any reason, the test is marked as failed (red) the Total Transaction time performance counter is not collected for that iteration. This implies that if the test always breaches the goals set, total transaction time is never collected. I consider this a bug and I have reported it to the Product group.

If transactions are used in web tests and a request inside a transaction fails you got the test location shown as unhealthy but the web test results green:

clip_image056[4]

Troubleshooting GSM

The GSM flow back and forth between the probes can be tracked using the Operations Manager event log on the GSM management server. The event source is “Health Service Modules Ex”, under normal circumstances the following events should be logged:

          10018 “Global Service Monitor Modules:  Successfully discovered VSWebTest container” where you can find the number of tests created

          10000 “Global Service Monitor Modules:  Successfully uploaded web tests to ‘https://gsm-prod.systemcenter.microsoft.com/TestConfigurationService’”

Other important events:

          10021 “Global Service Monitor Modules:  The test result file downloaded successfully” when a web test result file is downloaded from the cloud

The Download Web Test task needs read/write access to the temporary directory of the Default Action Account (normally LSA, thus c:\windows\temp) and to the Alert Attachments share. From what I could determine the web test result file is first downloaded in the temporary directory using the agent default action account and then moved to the alert attachments share via the alert attachments runas account.

Useful resources

How to Configure Integration with IntelliTrace Historical Profiling in System Center 2012 SP1

Map Dashboard in SC 2012 Operations Manager

Using the Web Application Availability Monitoring to monitor Web Applications health

System Center Global Service Monitor: Getting Started

Visual Studio Performance Testing Quick Reference Guide

  • Daniele

This posting is provided “AS IS” with no warranties, and confers no rights.

  1. #1 by Juan-Manuel Mora on July 10, 2015 - 9:40 pm

    how to build the script with a dynamic session IDs that change every time we run the test

  2. #2 by Jason on May 6, 2013 - 7:48 pm

    Greetings, I am trying to configure GSM but cannot even get the “Start Subscription” link to work. Our security team wants to know what firewall rules GSM needs, however I cannot find any documentation on what firewall rules are needed. Does anyone know what websites and ports the management servers need to have access to?

    • #3 by Daniele Grandini on May 7, 2013 - 7:18 am

      Hi Jason,
      the only port needed is tcp/443 for https communication to the MS cloud, in my post you can find the fqdn of the endpoints if you need to specify them. Don’t know why the link isn’t working for you, just tried and right now it works.

      Daniele

      • #4 by Jason on May 7, 2013 - 1:03 pm

        Thank you. The reason the link doesn’t work yet it it needs access to microsoftonline-p.com. I am working on getting the rules approved and created. Thank you for your assistance!

  1. Determining what is going on with GSM using OpsMgr event collection and views (#SCOM, #SYSCTR) | Catapult Systems
  2. Cameron Fuller

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: