[Dev Tip] Coded UI test on MTM


image10-33

Did you know you can manage and run CodedUI tests without Visual Studio? In an interview I recently did with .Net Rocks, one topic that came up was how to manage and run CodedUI tests on demand or as part of a nightly test run without using Visual Studio. You can accomplish this using Microsoft Test Manager.

ALM Test Infrastructure

One of the things that isn’t well known about the Microsoft ALM Test Infrastructure, which is part of the Visual Studio ALM tool suite, is the ability to manage your automated tests using the Microsoft Test Manager (MTM). With MTM you can manage your regression suites and run the regression suites straight from the tool. To give you some insight on how this is done, let’s walk through the steps involved to set this up.

First let’s assume you’ve built a larger set of CodedUI tests that exercise your application. Since the test set has become quite large, running all of the tests might take several hours to complete. How can you run the tests required to validate a change without running all of the tests every time? With MTM you have the ability to build a “test plan.” A test plan contains a set of test suites that can be static suites (you pick the tests that are part of the suite), requirement-based suites (you select the requirement that links to the test cases), or query-based suites (you created queries to select the tests that are part of the suite). With query-based suites you can build up suites that target a specific area of your application.

Here’s a scenario I like to use to explain how things play out. Imagine we work at an insurance company that has a web-based application it bought from a vendor. It contains different insurance products and business processes which are managed with the product. Let’s say it has policies for personal liability, car insurance and home insurance. It has different processes such as policy affirmation, claim settlement and adding a policy holder. Now let’s assume the vendor made a change to the claim settlement process. In this case you would like to run a regression set across all products for that particular process to check if something is broken.

Setting up the MTM test cases that relate to the CodedUI tests

Our goal is to create test suites which are tied to the CodedUI tests, and use these sub-selections of tests to complete a test run that validates a specific change made to the system. For this to work there are a couple of things you need to get in place:

  1. You need to have a way to differentiate the tests from each other based on the focus areas you have in your application.
  2. You need to import all the automated tests as MTM test cases so you can manage and run them from MTM.

So let’s look at step one first. You need to name your tests in such a way that you can build regression suites for specific purposes. How can we make this work? Write your CodedUI tests using the page object pattern so you have a good maintainable set of UI tests (you can find more info on this pattern in my course on CodedUI and on my blog here).

Next you need to import those tests into MTM so you can then build out the regression sets.

When you build the test scenarios based on the page objects, you need to ensure you can query on the tests in such a way that you can group tests together based on process or product type. So you need a naming convention. In this particular case it would make sense to say that you have a naming convention where you name the tests as follows:<ProcessName>_<ProductName>_TestScenario. So this would result in names like: PolicyAffirmation_CarInsurance_PersonAgeAbove60and PolicyAffirmation_CarInsurance_PersonAgeBelow20.

This way you can immediately tell which scenario the tests are validating, and at a later stage, build test suites that group tests based on parts of the name.

Once we have written the tests, the next step is to import the tests in MTM. You can do this manually by creating the test cases in MTM, then going over to Visual Studio to link them to the actual CodedUI test, but it’s a very cumbersome process. Luckily there’s an alternative! You can import the tests based on the assembly that is built, then generate the test cases in MTM. For this you need to use the tool called TCM.exe, which is part of the standard Visual Studio installation.

Let’s assume the assembly that contains the CodedUI tests is calledInsuranceRegressionTests.dll. In order to import the tests in Microsoft Test Manager, or better stated, in Team Foundation Server so they show up in Microsoft Test Manager, you need to run the following command line:

>tcm.exe testcase /import /collection:<your TFS Server collection name>  /teamproject:<your team project>  /storage: InsuranceRegressionTests.dll

Optionally, you can filter the test cases that are imported by specifying the /category option, and that will match only tests that have the category attribute specified on top of the test methods or class.

After running this command, you can now run a query in MTM and look at the test cases that were imported. This will look something like shown in the screenshot below:

Structuring your tests

So now that we’ve imported all of our tests into MTM, let’s structure the test now in such a way that we can select a group of tests to validate either a process or product. To do this you first need to create a new test plan:

Now we have a clean new plan that we can start organizing. Add two static test suites at the top level of the plan that dissect the groups created in process and product as shown here below:

The next step is to fill up the process suite with additional query-based suites. Here we are defining the set of tests based on the name in the test containing either the name of the process we need or the product we need. To define a suite that shows all test cases that test the process “Policy Affirmation” we create a query that looks at the title of the test and requires the text “PolicyAffirmation” to be in the title as show below:

And we can do the same for particular products as show below for product, car insurance:

After we have done this for all processes and products, the full regression suite looks like this:

Setting up the Lab Management machines

Now that we have the test plan complete and created nice buckets that focus on specific processes or products, we can now run the tests from MTM. This requires one additional step and that is to set up a lab environment. A lab environment contains the machines that will function as the web client machine running the actual tests. To create the environment, select the “Lab Canagement” tab in MTM.

Now that you’ve created a new environment, select a test machine you have available in your network. In my case the name of the test machine is “TestClient.”

Running your tests from MTM

Now that we have a lab machine available, we can run the tests from MTM. To do this go to the “Test” tab and there you can select the option to run the test.

Now you will get the dialog asking where you would like to run the tests.

This is where you need specify the build the tests are part of (the build containing the CodedUI tests) and the test environment you want to use. In this case, it’s the VSALM test environment we just created. The final thing to note is that the test setting specifies what the diagnostic data collectors will capture during the test run. This is where you can specify to use test settings that contain Intellitrace log capture, video capture and even audio capture on the target machine.

After you start a run, you will get a results view that shows the progress:

And when the run is finished you’ll get a summary of the run and a overall status report on the test plan.

Running the regression suites as part of the daily build process

One thing haven’t mentioned yet is the fact that we can use the existing test plan we created together with a lab management build. This is a build defined in Visual Studio where we can specify the lab environment we want to use (VSALM), the test plane we want to execute, and the build we will use that contains the test assemblies. You can schedule this build to run every night on the latest build of the CodedUI tests, and this way each morning when you get into the office you’ll already see a full summary like one shown above, based on a fully automated regression test that is scheduled and executed.

So with these steps you can manage your CodedUI tests from Microsoft Test Manager. You can create nice functional buckets of test suites by using the query based suites, can run your tests on existing machines just by adding them to the lab environment, and any functional tester can run the regression suites without ever knowing about Visual Studio.

###################################################################################

A coded UI test provides a mechanism to automatically execute and drive the application in much the same manner as a human sitting in front of a computer. It operates at the user interface layer and can be programmed to validate elements of the UI at various points during the test to confirm that the application is behaving properly.

Coded UI tests can be authored in C# or Visual Basic, and Visual Studio 2012 provides tools to help auto-generate much of this required code.

We will start by creating a simple Coded UI Test using the Coded UI Test Builder and adding some validation logic.

 

Setting Up the Sample Application

We will use a simple web application having one web form.
Create a test project

We will need a test project in which to house Coded UI test. Click File -> New -> Project, which will display the New Project dialog box. Select Visual C# -> Test -> Coded UI Test Project. Name the project “CodedUITestProject” and select “Add to solution” in the Solution drop-down. Click OK.
“Generate Code for Coded UI Test” will appear, providing options for generating test.

Choose the first option “Record actions, edit UI map or add assertions”. This option allows recording a Coded UI test from scratch by navigating through the application in the same manner that a user might.

The moment you choose above mentioned option Coded UI Test Builder appears.

Assuming you have launched the web form (calculate.aspx) in browser. Click the record button of the Test Builder. You can now begin recording the Coded UI test by using the application in same manner you would expect a user to. Type 20 in first textbox, then type 30 in second textbox, and click the Add button.

 

You can visually inspect the actions that the Test Builder has captured by clicking on Show recorded steps of the Test Builder.

At this point of time, validation logic can be added to confirm that the result of addition operation is correct. Before adding assertion, performed steps must be converted into source code. Do so by clicking on Generate Code.

A dialog will appear prompting you to name the method you want to create within coded UI test. Type “EnterDataAndClickAdd”; then click Add and Generate to resume building coded UI test.

You can now add assertion logic to validate the properties of one or more controls. The Test Builder allows you to easily select the control you want to validate. Do so by clicking the Add Assert and dragging the crosshair icon from the Test Builder on to the bottommost label next to Result of the form. As you hover over controls, you will notice that they become highlighted to indicate which control you are selecting. Once you have selected the bottommost label of the form, release your mouse button.

The properties for the lblResult label will be displayed.

For this test, we want to confirm that the number 50 (the sum of 20 and 30) is properly displayed in the textbox. In the list of properties for this control, you will see that the InnerText property has a value of 50. Highlight this row, then click Add an Assertion. “Add assertion” dialog will appear, allowing you to define the behavior of your assertion. Click the Comparator drop – down to examine your assertion choices. Accept the default value (AreEqual) for Comparator and the current value (50) for Comparison Value. You can also type message in case assertion fails .Click OK. The Test Builder will display a message indicating that your assertion has been added.

Click Generate Code from within the Test Builder to codify the assertion just added. A dialog will appear to name the method the will correspond to you assertion. Name the method “AddAssert” and click Add and Generate. The Test Builder now converts the assertion you defined into C# and insert this into test project.

Once you are finished, close the Test Builder, which will return you to Visual Studio.

Generated Code

In Solution Explorer, Open CodedUITest1.cs.

This is the main execution harness for the test, and calls all of the action and assertion methods you have defined earlier, as shown here:

[TestMethod]

public void CodedUITestMethod1()

{

  1. this.UIMap.EnterDataAndClickAdd();
  2. this.UIMap.AddAssert();

}

To better understand what each method is actually doing, you can examine the partial class file name UIMap.Desginer.cs. Right click on the EnterDataAndClickAdd method call and select “Go to definition”.

public void EnterDataAndClickAdd()

{

#region Variable Declarations

HtmlEdit uITxtFNoEdit = this.UIHttplocalhost52996CaWindow.UIHttplocalhost52996CaDocument.UITxtFNoEdit;

HtmlEdit uITxtSNoEdit = this.UIHttplocalhost52996CaWindow.UIHttplocalhost52996CaDocument.UITxtSNoEdit;

HtmlInputButton uIAddButton = this.UIHttplocalhost52996CaWindow.UIHttplocalhost52996CaDocument.UIAddButton;

#endregion

// Type ’20’ in ‘txtFNo’ text box

uITxtFNoEdit.Text = this.EnterDataAndClickAddParams.UITxtFNoEditText;

// Type ’30’ in ‘txtSNo’ text box

uITxtSNoEdit.Text = this.EnterDataAndClickAddParams.UITxtSNoEditText;

// Click ‘Add’ button

  1.            Mouse.Click(uIAddButton, new Point(40, 13));

}

This method is responsible for various actions, as defined by the action recorded earlier. It will put values in two textboxes and then click on Add.

Running Test

Launch the form in browser. Open Test Explorer by clicking TEST menu -> Windows -> Test Explorer. You need to build the solution to load all the tests in Test Explorer. Right click on CodedUITestMethod1 and select Run Selected Test.
Avoid using your mouse or keyboard while the text executes. If you have recorded test properly, the values 20 and 30 will be inserted into the textboxes, and Add button will be clicked. When finished, the test results will be displayed as shown

Creating Data-Driven Coded UI Test

Coded UI Test can be data-driven. So that it can be run multiple times with different data sources. Instead of having to implement a number of different tests permutations, we can instead bind the test to a data source and access the current data row for the test.

The different data sources are: CSV, Excel, Test Case, XML and SQL Express.

We will use XML as data source to make the Coded UI Test we created earlier; data-driven. Add an XML file to CodedUITestProject by the name “CalcData.xml” and put following xml in it.

 

xml version=”1.0″ encoding=”utf-8″ ?>

<DataContextData>

<DataContextRow>

<InputValue1>11</InputValue1>

<InputValue2>22</InputValue2>

<ExpectedAddAnswer>33</ExpectedAddAnswer>

</DataContextRow>

<DataContextRow>

<InputValue1>12</InputValue1>

<InputValue2>11</InputValue2>

<ExpectedAddAnswer>23</ExpectedAddAnswer>

</DataContextRow>

</DataContextData>

 

Now open the “CodedUITest1.cs” and add DataSource attribute to CodedUITestMethod1 and modify the code as shown below.

 

[TestMethod]

[DataSource(“Microsoft.VisualStudio.TestTools.DataSource.XML”, “|DataDirectory|\\CalcData.xml”, “DataContextRow”, DataAccessMethod.Sequential)]

public void CodedUITestMethod1()

{

  1. this.UIMap.EnterDataAndClickAddParams.UITxtFNoEditText = TestContext.DataRow[“InputValue1”].ToString();
  2. this.UIMap.EnterDataAndClickAddParams.UITxtSNoEditText = TestContext.DataRow[“InputValue2”].ToString();
  3. this.UIMap.EnterDataAndClickAdd();
  1. this.UIMap.AddAssertExpectedValues.UIItem50PaneInnerText = TestContext.DataRow[“ExpectedAddAnswer”].ToString();
  2. this.UIMap.AddAssert();

}

 

We use TextContext.DataRow to fetch row from the datasource. So this method will run for each row in datasource; putting the values of columns in respective controls along the way.

Now, launch the form in browser and run the test from Test Explorer. If no assertion fails, it should pass for all datarows.

Note: If you encounter error “The unit test adapter failed to connect to the data source or to read the data” during test run; Right click on “CalcData.xml” in Solution Explorer and select Properties and in Copy to Output Directory, choose either Copy always or Copy if newer.

Make some changes to XML file to make the test fail.

 

xml version=”1.0″ encoding=”utf-8″ ?>

<DataContextData>

<DataContextRow>

<InputValue1>11</InputValue1>

<InputValue2>22</InputValue2>

<ExpectedAddAnswer>33</ExpectedAddAnswer>

</DataContextRow>

<DataContextRow>

<InputValue1>12</InputValue1>

<InputValue2>11</InputValue2>

<ExpectedAddAnswer>33</ExpectedAddAnswer>

</DataContextRow>

</DataContextData>

 

Notice we have changed expected value for second row to 33, which is wrong (12 + 11 = 23). Now run the test again.
You can notice the test passed for first row but failed for second row.

Apart from recorded asserts, you can also add assertion logic by writing custom code. Now we will add some code to check that FirstTextbox should not be blank. Open UIMap.cs and add following code.

 

public partial class UIMap

{

public void AssertIfFNoIsBlank()

{

HtmlEdit uITxtFNoEdit = this.UIHttplocalhost52996CaWindow.UIHttplocalhost52996CaDocument.UITxtFNoEdit;

  1. Assert.AreNotEqual(string.Empty, uITxtFNoEdit.Text.Trim(), “First Number cannot be blank.”);

}

}

 

Now add a call to AssertIfFNoIsBlank to CodedUITest1.cs.

 

public void CodedUITestMethod1()

{

  1. this.UIMap.EnterDataAndClickAddParams.UITxtFNoEditText = TestContext.DataRow[“InputValue1”].ToString();
  2. this.UIMap.EnterDataAndClickAddParams.UITxtSNoEditText = TestContext.DataRow[“InputValue2”].ToString();
  3. this.UIMap.EnterDataAndClickAdd();
  1. this.UIMap.AssertIfFNoIsBlank();
  1. this.UIMap.AddAssertExpectedValues.UIItem50PaneInnerText = TestContext.DataRow[“ExpectedAddAnswer”].ToString();
  2. this.UIMap.AddAssert();

}
Again modify the xml file so that first textbox remain blank; to make this test fail.

xml version=”1.0″ encoding=”utf-8″ ?>

<DataContextData>

<DataContextRow>

<InputValue1></InputValue1>

<InputValue2>22</InputValue2>

<ExpectedAddAnswer>33</ExpectedAddAnswer>

</DataContextRow>

<DataContextRow>

<InputValue1>12</InputValue1>

<InputValue2>11</InputValue2>

<ExpectedAddAnswer>23</ExpectedAddAnswer>

</DataContextRow>

</DataContextData>

 

Run the test again.
You can verify the test passed for second row but it failed for first row because we modified the xml for the first value to be blank. Similarly more custom assertion logic can be added to verify the UI elements.

You can now maintain the CalcData.xml file within test project to add new rows of data. Those rows can be used for future test runs.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s