Good HTML email from .trx Test Results In Team Build 2008

March 11, 2011 1 comment

I struggle with mstest.exe and its various intricacies… The test results that Visual Studio 2008 produces are pretty good, but the viewer is not very user-friendly.

I wanted my team to have a more visible notice for our WatiN functional test runs. The default team system email was just too hard to extract the information we desired with just a glance. I put together a couple things to transform the single .trx file our TFS 2008 team build outputs into an email in your inbox. Below is an xslt file that sits along side the build definition:

<?xml version="1.0" encoding="utf-8"?>
<!-- modified; original by Gavin Stevens: http://geekswithblogs.net/gavin/archive/2009/08/25/134311.aspx -->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:vs="http://microsoft.com/schemas/VisualStudio/TeamTest/2006"
                xmlns:msxsl="urn:schemas-microsoft-com:xslt"
                xmlns:user="http://tempuri.org/lebobitz">

  <msxsl:script language="CSharp" implements-prefix="user">
    <![CDATA[  
        // credit Chris Doggett
        // http://stackoverflow.com/questions/840120/timespan-formatting
        public static string Pluralize(int n, string unit)
        {
	        if (string.IsNullOrEmpty(unit)) return string.Empty;

	        n = Math.Abs(n); // -1 should be singular, too

	        return unit + (n == 1 ? string.Empty : "s");
        }

        // credit Chris Doggett
        // http://stackoverflow.com/questions/840120/timespan-formatting
        public static string TimeSpanInWords(TimeSpan aTimeSpan)
        {
	        System.Collections.Generic.List<string> timeStrings = new System.Collections.Generic.List<string>();

	        int[] timeParts = new[] { aTimeSpan.Days, aTimeSpan.Hours, aTimeSpan.Minutes, aTimeSpan.Seconds };
	        string[] timeUnits = new[] { "day", "hour", "minute", "second" };

	        for (int i = 0; i < timeParts.Length; i++)
	        {
		        if (timeParts[i] > 0)
		        {
			        timeStrings.Add(string.Format("{0} {1}", timeParts[i], Pluralize(timeParts[i], timeUnits[i])));
		        }
	        }

	        return timeStrings.Count != 0 ? string.Join(", ", timeStrings.ToArray()) : "0 seconds";
        }

        public string PrettyDateDifference(string start, string end)
        {
	        var startDate = DateTime.Parse(start);
	        var endDate = DateTime.Parse(end);
	        TimeSpan ts = endDate.Subtract(startDate);
	        return TimeSpanInWords(ts);
        }
        
        public string PrettyDate(string date)
        {
          DateTime d = DateTime.Parse(date);
          return string.Format("{0} {1}", d.ToLongDateString(), d.ToLongTimeString());
        }
    ]]>
  </msxsl:script>

  <xsl:template match="/">
    <html>
      <head>
        <style type="text/css">
          
        </style>
      </head>
      <body style="font-family:Verdana; font-size:10pt">
        <h1>Test Results Summary</h1>
        <table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
          <tr>
            <td width="150">
              <b>Start Date/Time</b>
            </td>
            <td width="550">
              <xsl:value-of select="user:PrettyDate(//vs:Times/@start)"/>
            </td>
          </tr>
          <tr>
            <td width="150">
              <b>End Date/Time</b>
            </td>
            <td width="550">
              <xsl:value-of select="user:PrettyDate(//vs:Times/@finish)"/>
            </td>
          </tr>
          <tr>
            <td width="150">
              <b>Total Time</b>
            </td>
            <td width="550">
              <xsl:value-of select="user:PrettyDateDifference(//vs:Times/@start,//vs:Times/@finish)"/>
            </td>
          </tr>
          <tr>
            <td width="150">
              <b>Results</b>
            </td>
            <td width="550">
              <xsl:value-of select="//vs:TestRun/@name"/>
            </td>
          </tr>
        </table>
        <xsl:call-template name="summary" />
        <xsl:call-template name="details" />
      </body>
    </html>
  </xsl:template>
  <xsl:template name="summary">
    <h3>Test Summary</h3>
    <table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt">
      <tr>
        <td style="font-weight:bold">Total</td>
        <td style="font-weight:bold">Failed</td>
        <td style="font-weight:bold">Passed</td>
      </tr>
      <tr>
        <td >
          <xsl:value-of select="//vs:ResultSummary/vs:Counters/@total"/>
        </td>
        <td style="background-color:pink;">
          <xsl:value-of select="//vs:ResultSummary/vs:Counters/@failed"/>
        </td>
        <td style="background-color:lightgreen;">
          <xsl:value-of select="//vs:ResultSummary/vs:Counters/@passed"/>
        </td>
      </tr>
    </table>
  </xsl:template>
  <xsl:template name="details">
    <h3>Test Details</h3>
    <table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
      <tr>
        <td style="font-weight:bold">Test Name</td>
        <td style="font-weight:bold">Result</td>
        <td style="font-weight:bold">Duration</td>
      </tr>
      <xsl:for-each select="//vs:Results/vs:UnitTestResult">
		<xsl:sort select="@outcome" order="ascending" />
        <xsl:variable name="executionId">
          <xsl:value-of select="@executionId"/>
        </xsl:variable>
        <tr>
          <xsl:attribute name="style">
            <xsl:choose>
              <xsl:when test="@outcome = 'Failed'">background-color:pink;</xsl:when>
              <xsl:when test="@outcome = 'Passed'">background-color:lightgreen;</xsl:when>
              <xsl:otherwise>background-color:yellow;</xsl:otherwise>
            </xsl:choose>
          </xsl:attribute>
          <td>
            <xsl:value-of select="@testName"/>
          </td>
          <td>
            <xsl:choose>
              <xsl:when test="@outcome = 'Failed'">FAILED</xsl:when>
              <xsl:when test="@outcome = 'Passed'">Passed</xsl:when>
              <xsl:otherwise>Inconclusive</xsl:otherwise>
            </xsl:choose>
          </td>
          <td>
            <xsl:value-of select="@duration"/>
          </td>
        </tr>
        <tr>
          <td colspan="3">
			  <xsl:choose>
              <xsl:when test="@outcome = 'Failed'">
					<div  id="{$executionId}">
					  <table style="width:700;border:1px solid black;font-family:Verdana; font-size:10pt;">
							  <tr>
								<td width="10"></td>
								<td>
								  <pre style="word-wrap: break-word">
									<xsl:value-of select="vs:Output/vs:ErrorInfo/vs:Message"/>
								  </pre>
								</td>
							  </tr>
							  <tr>
								<td width="10"></td>
								<td>
								  <pre style="word-wrap: break-word">
									<xsl:value-of select="vs:Output/vs:ErrorInfo/vs:StackTrace"/>
								  </pre>
								</td>
							  </tr>
					  </table>
					</div>
			  </xsl:when>
              <xsl:otherwise></xsl:otherwise>
            </xsl:choose>
          </td>
        </tr>
      </xsl:for-each>
</table>
  </xsl:template>
</xsl:stylesheet>

And then in the TfsBuild.proj itself, there are some customizations:

...
<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Communication.Email" />
  <UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Xml.XmlTask" />
  <UsingTask AssemblyFile="$(MSBuildExtensionsPath)\ExtensionPack\MSBuild.ExtensionPack.dll" TaskName="MSBuild.ExtensionPack.Framework.MsBuildHelper" />
  
...
 <Target Name="AfterTest" DependsOnTargets="ParseTrxAndEmail;">
  </Target>

  <Target Name="ParseTrxAndEmail">
    <CreateItem Include="$(TestResultsRoot)\*.trx">
      <Output TaskParameter="Include" ItemName="TrxFiles" />
    </CreateItem>

    <MSBuild.ExtensionPack.Framework.MsBuildHelper TaskAction="GetItem" InputItems1="@(TrxFiles)" Position="0">
      <Output TaskParameter="OutputItems" ItemName="TheTrx" />
    </MSBuild.ExtensionPack.Framework.MsBuildHelper>

    <MSBuild.ExtensionPack.Xml.XmlTask TaskAction="Transform"
                                       Conformance="Document"
                                       XslTransformFile="TrxTransform.xslt"
                                       XmlFile="%(TheTrx.FullPath)">

      <Output PropertyName="TrxTransformed" TaskParameter="Output"/>
    </MSBuild.ExtensionPack.Xml.XmlTask>
    <MSBuild.ExtensionPack.Communication.Email TaskAction="Send" Subject="$(RequestedFor)'s WatiN tests for $(MainBuildLabelToDeploy)" SmtpServer="mail.tempuri.org" MailFrom="buildservice@tempuri.org" MailTo="lebobitz@tempuri.org" Body="$(TrxTransformed)" />
  </Target>
...
Categories: .Net, MsBuild, WatiN

Synchronizing WatiN and Ajax with jQuery

March 6, 2011 6 comments

WatiN (pronounced “what in”) is an open source .Net project that allows the programmatic automation of browsers (IE/Firefox/Chrome).  My company uses WatiN within Visual Studio unit tests to automate the testing of our web application.  It works interactively with the browser window: clicking, entering text, and navigating just like a user.   With the unit tests, assertions can be made about the state of the application through various user scenarios.  The project is under active development on sourceforge.  In short, it’s a wonderful tool and isn’t hard to use.

We maintain hundreds of test cases against our web application.  We chain team builds so that when the app build finishes building successfully, it kicks off another build which ships and runs mstest against the WatiN solution.  There’s a dedicated build agent running in interactive mode which runs the tests.  I’ll cover this setup in a future post.

WatiN abstracts the browser/DOM/javascript engine into its own API.  It provides a consistent abstraction for several major browsers.  At a high level, it works by instantiating the browser under test and holding a native windows handle to the browser window.  Then unmanaged calls into User32.dll are used to perform various actions on the browser.

There’s a good amount of synchronization that is done natively by WatiN.  For example, when you click a link that goes to a new page, the click is simulated and then the WatiN code waits for the browser to be ready when the next page is loaded.  In most cases this works perfectly.  Here’s an example from the WatiN homepage:

[Test]
public void SearchForWatiNOnGoogle()
{
  using (var browser = new IE("http://www.google.com"))
  {
    browser.TextField(Find.ByName("q")).TypeText("WatiN");
    browser.Button(Find.ByName("btnG")).Click();

    Assert.IsTrue(browser.ContainsText("WatiN"));
  }
}

Between the .Click() method and the Assert, WatiN waits for the DOM to complete loading.  See the DomContainer.WaitForComplete() method in WatiN to understand how this works.

However, modern web applications change their pages with Ajax in addition to full page loads.  This can been a hairy problem for every browser automation library, including WaitN.  WatiN doesn’t know which page interaction will trigger Ajax behavior, and the page may be different after the interaction.  Another synchronization mechanism in needed.

If you are using jQuery .ajax, you’re in luck.

jQuery has global Ajax event handlers ajaxStart and ajaxStop which wrap all ajax activity.  Any time Ajax requests start, ajaxStart is called; and after all ajax requests have finished, ajaxStop is called.  We can write a piece of code in a central place (common js or masterpage) to keep track of whether the page is currently in the middle of an ajax request.

// used to syncronize the WaitN tests
var isRunningAjax = false;
$(document).ajaxStop(function() { isRunningAjax = false; });
$(document).ajaxStart(function() { isRunningAjax = true; });

Now we’ll give WatiN  the ability to poll the page for the isRunningAjax variable, using its ability to run any javascript with the Browser.Eval method.  We do this with a simple extension method on Browser, which is written in the same spirit as WatiN’s native WaitForComplete():


internal static void WaitForAjaxComplete(this DomContainer browser)  {            
  WaitForAjaxComplete(browser, 150);        
}

internal static void WaitForAjaxComplete(this DomContainer browser, int pollingIntervalInMs) {
  while (true)
  {                
    Thread.Sleep(pollingIntervalInMs);                                    
    if(browser.Eval("typeof(isRunningAjax)") == "undefined" || browser.Eval("isRunningAjax") == "false") { 
      break;
    }
  }        
}

Normally, a while(true) would be a no-no in production code, but in this case since the mstest runner has a timeout (default 30s) for the test to run, the worst that can happen is 30 seconds of waiting followed by a failed test. Also, notice that typeof(isRunningAjax) makes sure the synchronization mechanism is on the page (in case you forget to put it there, or another js error prevented its evaluation).

Finally, here’s a snippet of WatiN code that takes advantage of our new synchronization for jQuery Ajax calls:

 Browser.TextField(Find.ById(SOME_ID)).TypeText(SOME_SEARCH_KEY);
 Browser.WaitForAjaxComplete();
 Div div = Browser.Div(Find.ByClass("savedResultsBlock"));
 div.Link(Find.ByTitle(SOME_VALUE)).Click();

I’ve been in awe of WatiN for a while now and I’m planning some more posts describing how we have set up a page/control wrapper to WatiN that provides an even higher layer of abstraction so that the test methods can be written against an Api specific to our application.

Categories: .Net, jQuery, Uncategorized, WatiN

Net.Msmq Wcf Binding and ANONYMOUS USER permission

January 19, 2011 Leave a comment

My boss was frustrated after a weekend trying to track down a problem we were having with WCF’s Net.Msmq binding. He stumbled on the solution to the permission issue this morning (after reading many blogs without help).

Symptom

WCF 5.0 on Windows 7. Using Net.Msmq binding and a remote queue. No client credentials being used. Machines within the same domain. Getting Access Denied error being sent back from the service. SysInternals Process Monitor and event logs did no provide any clue to the problem. Network Service and CLIENTMACHINE$ both had full permissions.

Solution

ANONYMOUS USER permission for Send, Read, and Peak Messages was required on the machine hosting the service for the message to be processed on the service side.

My boss worked backwards to the solution when he saw it work after giving Everyone full permissions on the queue.

Lesson Learned

always try Everyone full permissions when encountering a similar problem to rule out security being an issue. Then work back to the smallest set of credentials that are required. We can afford to use ANONYMOUS USER in our case because both machines are behind the same firewall in our production environment.

Categories: Net.Msmq, Troubleshooting, WCF

Hello world!

January 16, 2011 Leave a comment

I’m a software developer who reads a lot of technology blogs, but I haven’t written content myself.

I started developing professionally in 2003 after graduating college with degrees in CS and math.  I’ve been working the past 7 years for a prominent political consultant in Washington, DC.  My company’s been using the .Net framework since version 1.1.  There have been so many exciting developments on the Windows stack since I started.

As a developer, I’ve found blogs to be an invaluable source of tips, techniques, patterns, workarounds, and inspiration in my career (particularly in the .Net realm).  My reason for this blog is to give back some of the same to the blogosphere.  I’m hoping to stay mostly technical and provide snippets, workarounds, and reviews of some of my favorite patterns and practices.  My goal for 2011 is a slow, yet consistent, start.  I’m shooting to write at least one substantive post a month.

Craig Lebowitz

Categories: Uncategorized
Follow

Get every new post delivered to your Inbox.