SpecFlow.exe and MsTest

With your SpecFlow installation comes SpecFlow.exe that is a program that can be used to generate tests from the scenarios AND to create nicely formatted reports from the generated test results.
There's been a lot written on how to generate these reports when your using NUnit (see this and this for example), but when it comes to managing this for MsTest there's been almost silent. And facing this problem I can sure see why... It's a bit trickier.

In this blog post I want to show you two things; how to generate MsTest's from your .feature-files and how to create a report from the generated results. Finally I'll show you how to put the two together and link it up to a nice "External tool"-button in Visual Studio. Here we go:

Generate MsTest's from your .feature-file

With this step you can generate the test from you scenarios. SpecFlow.exe picks up your configuration and generates the test in your test framework of choice.
From the look of it, it seems quite straight forward. Using the “help command” of SpecFlow.exe, “specflow help generateall” produces this help:

Generate tests from all feature files in a project
usage: specflow generateall projectFile [/force] [/verbose]
    projectFile  Visual Studio Project File containing features

OK. I put together a .bat file that does that. (Note that I’m on a 64-bit machine and had to use some funky DOS-shortcut to get there. No simple way to be architecture agnostic I’m afraid).
Here is my file:

"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" 
generateAll Specs\Specs.csproj  
/force /verbose

pause

I’ve added some line breaks for readability.

Well – that was easy.

Create a report from the generated results

To get this step to work we have to run the test and get hold of the location of the test report file (.trx). When you do this from within Visual Studio the test reporting is done in a TestResult folder and the file get a name with a timestamp. That is not very script-friendly sadly and we’re forced into writing a bat-file that also run the tests.

I used the MsTest Command line reference to put together this .bat-file:

if Exist TestResult.trx del TestResult.trx 

"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" 

/testcontainer:Specs\bin\Debug\Specs.dll 

/resultsfile:TestResult.trx 

pause

Again, watch out for me using a 64-bit Windows and use %programfiles% if you’re not.

Some strangeness with MsTest made me delete the testResults.trx before each run. Also MsTest create some folders (username_testrun...) but that doesn’t bother me now.

When we now have a created .trx file we can run the command that creates the report for us. According to the “documentation” (specflow help mstestexecutionreport"):

usage: specflow mstestexecutionreport projectFile 
[/testResult:value] 
[/xsltFile:value] 
[/out:value]

• The projectFile is the Visual Studio Project File containing features and specifications.
  
• Test Result refers to the .trx file generated by MsTest And it defaults to TestResult.trx
  
• Xslt file to use, defaults to built-in stylesheet if not provided
  
• out is the generated output file. Defaults to TestResult.html

I’m happy with the defaults of the last three since I choose my .trx-file name ... wisely. :)
So my whole .bat-file becomes this:

"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport Specs\Specs.csproj 

pause

A low and behold; it actually produces the nice report we wanted:

Putting it all together

That’s neat – we now have three different .bat files that we need to click in consecutive order ;)
No really – the first one (generate test from features can most certainly be handled by Visual Studio in most cases. Or in any case will probably not run in conjunction with the other two steps.
But to run the tests and produce a report would be nice with a single file. Here it is:

if Exist TestResult.trx del TestResult.trx 

"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:Specs\bin\Debug\Specs.dll /resultsfile:TestResult.trx 

"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport Specs\Specs.csproj /testResult:TestResult.trx

pause

And looking to this blog post I’ve also created a parameterized version of it that I can hook up to a “external command” that does that with a single click. That changes the bat-file into this:

if Exist TestResult.trx del TestResult.trx 

"%ProgramFiles(x86)%\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:%2 /resultsfile:TestResult.trx 

"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" mstestexecutionreport %1 /testResult:TestResult.trx /out:TestResult.html

echo Created file TestResult.html

So you need to send the name of the test container (the .dll) and the project file. Save that file to a known location so that you can point your external command to it.
Finally you can create a external tool button in Visual Studio and set the parameters as follows:

The arguments to the external command are:

• $(ProjectDir)$(ProjectFileName)
  
• $(TargetName)$(TargetExt)

Note that the project with specifications has to be selected before the External command can be run.

Conclusion

In this blog post I’ve learned how hook up SpecFlow to generate tests and reports for me.
You can download my code here