Archive
Good HTML email from .trx Test Results In Team Build 2008
I struggle with mstest.exe and its various intricacies… The test results that Visual Studio 2008 produces are pretty good, but the viewer is not very user-friendly.
I wanted my team to have a more visible notice for our WatiN functional test runs. The default team system email was just too hard to extract the information we desired with just a glance. I put together a couple things to transform the single .trx file our TFS 2008 team build outputs into an email in your inbox. Below is an xslt file that sits along side the build definition:
<?xml version="1.0" encoding="utf-8"?>
<!-- modified; original by Gavin Stevens: http://geekswithblogs.net/gavin/archive/2009/08/25/134311.aspx -->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:vs="http://microsoft.com/schemas/VisualStudio/TeamTest/2006"
xmlns:msxsl="urn:schemas-microsoft-com:xslt"
xmlns:user="http://tempuri.org/lebobitz">
<msxsl:script language="CSharp" implements-prefix="user">
<![CDATA[
// credit Chris Doggett
// http://stackoverflow.com/questions/840120/timespan-formatting
public static string Pluralize(int n, string unit)
{
if (string.IsNullOrEmpty(unit)) return string.Empty;
n = Math.Abs(n); // -1 should be singular, too
return unit + (n == 1 ? string.Empty : "s");
}
// credit Chris Doggett
// http://stackoverflow.com/questions/840120/timespan-formatting
public static string TimeSpanInWords(TimeSpan aTimeSpan)
{
System.Collections.Generic.List<string> timeStrings = new System.Collections.Generic.List<string>();
int[] timeParts = new[] { aTimeSpan.Days, aTimeSpan.Hours, aTimeSpan.Minutes, aTimeSpan.Seconds };
string[] timeUnits = new[] { "day", "hour", "minute", "second" };
for (int i = 0; i < timeParts.Length; i++)
{
if (timeParts[i] > 0)
{
timeStrings.Add(string.Format("{0} {1}", timeParts[i], Pluralize(timeParts[i], timeUnits[i])));
}
}
return timeStrings.Count != 0 ? string.Join(", ", timeStrings.ToArray()) : "0 seconds";
}
public string PrettyDateDifference(string start, string end)
{
var startDate = DateTime.Parse(start);
var endDate = DateTime.Parse(end);
TimeSpan ts = endDate.Subtract(startDate);
return TimeSpanInWords(ts);
}
public string PrettyDate(string date)
{
DateTime d = DateTime.Parse(date);
return string.Format("{0} {1}", d.ToLongDateString(), d.ToLongTimeString());
}
]]>
</msxsl:script>
<xsl:template match="/">
<html>
<head>
<style type="text/css">
</style>
</head>
<body style="font-family:Verdana; font-size:10pt">
<h1>Test Results Summary</h1>
<table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
<tr>
<td width="150">
<b>Start Date/Time</b>
</td>
<td width="550">
<xsl:value-of select="user:PrettyDate(//vs:Times/@start)"/>
</td>
</tr>
<tr>
<td width="150">
<b>End Date/Time</b>
</td>
<td width="550">
<xsl:value-of select="user:PrettyDate(//vs:Times/@finish)"/>
</td>
</tr>
<tr>
<td width="150">
<b>Total Time</b>
</td>
<td width="550">
<xsl:value-of select="user:PrettyDateDifference(//vs:Times/@start,//vs:Times/@finish)"/>
</td>
</tr>
<tr>
<td width="150">
<b>Results</b>
</td>
<td width="550">
<xsl:value-of select="//vs:TestRun/@name"/>
</td>
</tr>
</table>
<xsl:call-template name="summary" />
<xsl:call-template name="details" />
</body>
</html>
</xsl:template>
<xsl:template name="summary">
<h3>Test Summary</h3>
<table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt">
<tr>
<td style="font-weight:bold">Total</td>
<td style="font-weight:bold">Failed</td>
<td style="font-weight:bold">Passed</td>
</tr>
<tr>
<td >
<xsl:value-of select="//vs:ResultSummary/vs:Counters/@total"/>
</td>
<td style="background-color:pink;">
<xsl:value-of select="//vs:ResultSummary/vs:Counters/@failed"/>
</td>
<td style="background-color:lightgreen;">
<xsl:value-of select="//vs:ResultSummary/vs:Counters/@passed"/>
</td>
</tr>
</table>
</xsl:template>
<xsl:template name="details">
<h3>Test Details</h3>
<table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
<tr>
<td style="font-weight:bold">Test Name</td>
<td style="font-weight:bold">Result</td>
<td style="font-weight:bold">Duration</td>
</tr>
<xsl:for-each select="//vs:Results/vs:UnitTestResult">
<xsl:sort select="@outcome" order="ascending" />
<xsl:variable name="executionId">
<xsl:value-of select="@executionId"/>
</xsl:variable>
<tr>
<xsl:attribute name="style">
<xsl:choose>
<xsl:when test="@outcome = 'Failed'">background-color:pink;</xsl:when>
<xsl:when test="@outcome = 'Passed'">background-color:lightgreen;</xsl:when>
<xsl:otherwise>background-color:yellow;</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
<td>
<xsl:value-of select="@testName"/>
</td>
<td>
<xsl:choose>
<xsl:when test="@outcome = 'Failed'">FAILED</xsl:when>
<xsl:when test="@outcome = 'Passed'">Passed</xsl:when>
<xsl:otherwise>Inconclusive</xsl:otherwise>
</xsl:choose>
</td>
<td>
<xsl:value-of select="@duration"/>
</td>
</tr>
<tr>
<td colspan="3">
<xsl:choose>
<xsl:when test="@outcome = 'Failed'">
<div id="{$executionId}">
<table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
<tr>
<td width="10"></td>
<td>
<pre style="word-wrap: break-word">
<xsl:value-of select="vs:Output/vs:ErrorInfo/vs:Message"/>
</pre>
</td>
</tr>
<tr>
<td width="10"></td>
<td>
<pre style="word-wrap: break-word">
<xsl:value-of select="vs:Output/vs:ErrorInfo/vs:StackTrace"/>
</pre>
</td>
</tr>
</table>
</div>
</xsl:when>
<xsl:otherwise></xsl:otherwise>
</xsl:choose>
</td>
</tr>
</xsl:for-each>
</table>
</xsl:template>
</xsl:stylesheet>
And then in the TfsBuild.proj itself, there are some customizations:
...
<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Communication.Email" />
<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Xml.XmlTask" />
<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Framework.MsBuildHelper" />
...
<Target Name="AfterTest" DependsOnTargets="ParseTrxAndEmail;">
</Target>
<Target Name="ParseTrxAndEmail">
<CreateItem Include="$(TestResultsRoot)\*.trx">
<Output TaskParameter="Include" ItemName="TrxFiles" />
</CreateItem>
<MSBuild.ExtensionPack.Framework.MsBuildHelper TaskAction="GetItem" InputItems1="@(TrxFiles)" Position="0">
<Output TaskParameter="OutputItems" ItemName="TheTrx" />
</MSBuild.ExtensionPack.Framework.MsBuildHelper>
<MSBuild.ExtensionPack.Xml.XmlTask TaskAction="Transform"
Conformance="Document"
XslTransformFile="TrxTransform.xslt"
XmlFile="%(TheTrx.FullPath)">
<Output PropertyName="TrxTransformed" TaskParameter="Output"/>
</MSBuild.ExtensionPack.Xml.XmlTask>
<MSBuild.ExtensionPack.Communication.Email TaskAction="Send" Subject="$(RequestedFor)'s WatiN tests for $(MainBuildLabelToDeploy)" SmtpServer="mail.tempuri.org" MailFrom="buildservice@tempuri.org" MailTo="lebobitz@tempuri.org" Body="$(TrxTransformed)" />
</Target>
...
Synchronizing WatiN and Ajax with jQuery
WatiN (pronounced “what in”) is an open source .Net project that allows the programmatic automation of browsers (IE/Firefox/Chrome). My company uses WatiN within Visual Studio unit tests to automate the testing of our web application. It works interactively with the browser window: clicking, entering text, and navigating just like a user. With the unit tests, assertions can be made about the state of the application through various user scenarios. The project is under active development on sourceforge. In short, it’s a wonderful tool and isn’t hard to use.
We maintain hundreds of test cases against our web application. We chain team builds so that when the app build finishes building successfully, it kicks off another build which ships and runs mstest against the WatiN solution. There’s a dedicated build agent running in interactive mode which runs the tests. I’ll cover this setup in a future post.
WatiN abstracts the browser/DOM/javascript engine into its own API. It provides a consistent abstraction for several major browsers. At a high level, it works by instantiating the browser under test and holding a native windows handle to the browser window. Then unmanaged calls into User32.dll are used to perform various actions on the browser.
There’s a good amount of synchronization that is done natively by WatiN. For example, when you click a link that goes to a new page, the click is simulated and then the WatiN code waits for the browser to be ready when the next page is loaded. In most cases this works perfectly. Here’s an example from the WatiN homepage:
[Test]
public void SearchForWatiNOnGoogle()
{
using (var browser = new IE("http://www.google.com"))
{
browser.TextField(Find.ByName("q")).TypeText("WatiN");
browser.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(browser.ContainsText("WatiN"));
}
}
Between the .Click() method and the Assert, WatiN waits for the DOM to complete loading. See the DomContainer.WaitForComplete() method in WatiN to understand how this works.
However, modern web applications change their pages with Ajax in addition to full page loads. This can been a hairy problem for every browser automation library, including WaitN. WatiN doesn’t know which page interaction will trigger Ajax behavior, and the page may be different after the interaction. Another synchronization mechanism in needed.
If you are using jQuery .ajax, you’re in luck.
jQuery has global Ajax event handlers ajaxStart and ajaxStop which wrap all ajax activity. Any time Ajax requests start, ajaxStart is called; and after all ajax requests have finished, ajaxStop is called. We can write a piece of code in a central place (common js or masterpage) to keep track of whether the page is currently in the middle of an ajax request.
// used to syncronize the WaitN tests
var isRunningAjax = false;
$(document).ajaxStop(function() { isRunningAjax = false; });
$(document).ajaxStart(function() { isRunningAjax = true; });
Now we’ll give WatiN the ability to poll the page for the isRunningAjax variable, using its ability to run any javascript with the Browser.Eval method. We do this with a simple extension method on Browser, which is written in the same spirit as WatiN’s native WaitForComplete():
internal static void WaitForAjaxComplete(this DomContainer browser) {
WaitForAjaxComplete(browser, 150);
}
internal static void WaitForAjaxComplete(this DomContainer browser, int pollingIntervalInMs) {
while (true)
{
Thread.Sleep(pollingIntervalInMs);
if(browser.Eval("typeof(isRunningAjax)") == "undefined" || browser.Eval("isRunningAjax") == "false") {
break;
}
}
}
Normally, a while(true) would be a no-no in production code, but in this case since the mstest runner has a timeout (default 30s) for the test to run, the worst that can happen is 30 seconds of waiting followed by a failed test. Also, notice that typeof(isRunningAjax) makes sure the synchronization mechanism is on the page (in case you forget to put it there, or another js error prevented its evaluation).
Finally, here’s a snippet of WatiN code that takes advantage of our new synchronization for jQuery Ajax calls:
Browser.TextField(Find.ById(SOME_ID)).TypeText(SOME_SEARCH_KEY);
Browser.WaitForAjaxComplete();
Div div = Browser.Div(Find.ByClass("savedResultsBlock"));
div.Link(Find.ByTitle(SOME_VALUE)).Click();
I’ve been in awe of WatiN for a while now and I’m planning some more posts describing how we have set up a page/control wrapper to WatiN that provides an even higher layer of abstraction so that the test methods can be written against an Api specific to our application.
Lebobitz 