Executing Coded UI Tests from Quality Center

A few months ago I was exploring the features and practicalities of using Coded UI for automation. One consideration to take into account was whether or not coded UI tests could be executed from Quality Center. 

First of all, why would you want to do this? Well, the QA process at the client site was all geared around Quality Center - test cases, metrics, defects etc so to go down a pure CodedUI route would mean switching all tooling and processes over to TFS. If you've worked in large organisations you'll appreciate that's a fairly long roadmap! 

An iterim solution to was to bridge this gap by seeing if we could execute CodedUI tests from Quality Center.

In this article I will show you the proof of concept I managed to implement. We never went down the CUI route so I haven't been able to spend any more time refining this, but it will get you off the ground and help you explore the possibilities around this integration. This article assumes you have some experience using Visual Studio and Coded UI.

The goals

In order for this to work, the following criteria needs to be supported by the solution:

1. CUI Test Cases must be represented in Quality Center.
2. CUI Test Cases must be launched by the automatic runner featur
3. The execution status of the test must be captured.
4. A meaningful execution report / logfile must be attached to the results to support analysis.

The limitations

There were a couple of design factors to take into account:

1. Our CUI tests were designed on a 1 to 1 basis, i.e 1 CUI test represented 1 test case. Multiple iterations were not factored into the design. If your CUI test does use multiple iterations, then you will need to adjust the results analysis function to determine the overall run status for the test. There may also be some clever ways to change the command line execution to run specific iterations.
2. A QC remote agent feature is not supported so an instance of Quality Center needs to be run on each machine where you are executing the test. I suggest using the standalone QC client exe (the one that doesn't use IE) so that CUI doesn't get confused with the QC browser.

Implementation

To make this article easier to digest, I've split it up into 4 main stages:

1. Installing mstest.exe on the client machines.
2. Compiling the CUI test and placing the dll on the client machine.
3. Creating a VAPI-XP test in Quality Center
4. Executing the CUI test and analysing the results

1 - Installing mstest.exe on the client machines

mstest.exe is required to execute the coded ui tests, it's a command line tool which you use to launch the tests. We will invoke mstest.exe from Quality Center and tell it which coded ui test to execute.

You will need to install the agent as an interactive process. This Microsoft Article explains more. If you don't have the CD it looks like you can download the agent here.

2 - Compiling the CUI test and placing the dll on the client machine

The next step is to compile your tests and save them onto the client machine. The basics of how to do this are as follows:

1. Load your CUI project in Visual Studio.
2. Right click the project and select “build”.
3. In the output window, note the directory that the dll is compiled to.
4. Copy the entire directory to the the host machine where you will be executing the test on(ensure you place it on the C: drive of the host machine, this may have been a system policy in place where I worked but I couldn't launch the tests from any other drive).

Note: If you plan to run a test on a machine where Visual Studio is not installed, you will need to compile it with all of the supporting dll's copied into the local folder, as per the screenshot below:

3 - Create a VAPI-XP test in the Quality Center test plan

You will need to create a test in Quality Center that represents the CUI Test. This is so that you can run it from the test lab. To do this, create a VAPI-XP test and then write some vbscript code that calls the CUI test:

1) Navigate to the QC Test Plan

2) Select "Tests" > "New Test".

    - Type: VAPI-XP Test

    - Name: Give it a relavent name

    - Click OK

3) Choose the Script Language as "VBScript" then click Next,

4) Set the test type to "Console Application Test" then click Next.

5) Click on Finish.

6) Select the test and click on the "Test Script" tab.

7) Replace the vbscript code with this code.

8) Now you will need to modify the code. The following lines of code will need to be changed in accordance with your setup:

• Dim strPathMSTestExe :   strPathMSTestExe = "<path to where mstest.exe is installed>"
• Dim strTestContainerDLL : strTestContainerDLL = "<path to the compiled test dll>"
• Dim strResultsDirectory : strResultsDirectory = "<path to a directory where the results can be saved>"
• Dim strTestName : strTestName = "<the name of the coded ui test to run>"  

Once you've done this, the test is ready to execute from the test lab.

4 - Executing the CUI Test and Analysing the Results

To execute the test, simply add it to a Quality Center test set and run it locally (I'm assuming that having come this far you can manage this!).

Remember the limitation we have is that it can only be run locally (unless you want to program some sort of agent yourself). Therefore I recommend running QC from the standalone client so that the CUI test doesn't get confused with any QC IE sessions that may be open.

I have programmed a few functions to analyse the results (the .trx file), retrieve any attachments and upload them to the QC. These are attached to each individual run .The functions performing this are called:

• GetResultsXMLStatus
• AttachFileToResults
• GetArrayOfAttachments

These can be found at the bottom of the vbscript code.

Note: This method is based on our setup that one test equals one iteration. If you run multiple iterations within a single test then you may need to implement your own way of analysing the results.

Closing Comments

This article demonstrates a proof of concept that CUI tests can be executed through Quality Center. Unfortunately I'm no longer engaged on a project using CodedUI so I haven't been able to take this further.

I welcome any comments, useful suggestions or enhancements you've been able to make. Hopefully this should give you enough help to get off the ground!