Blog

NUnit vs Team System Unit Test

May 15, 2007

At work there's a discussion going on whether we will choose NUnit or Team System Unit Test for our unit tests. Currently we have about 1000+ unit tests written in NUnit and we are most pleased about it. The reason that this discussion has come up is a company-wide adoption of unit testing, something that we are all very happy about. 

In order to get a clear and objective view, I gathered all pros and cons for each of these unit test frameworks. Here goes ...

First I made a listing of all attributes:

NUnit attributes TS Unit Test attributes
[Category(“name”)] [AssemblyCleanup]
[Description(“description”)] [AssemblyInitialize]
[ExpectedException(“myException”)] [ClassCleanup]
[Explicit] [ClassInitialize]
[Ignore(“reason”)] [CssIteration("name")]
[Platform(“name”)] [TestProperty("name", "value")]
[Property(“name”, “value”)] [CssProjectStructure("name")]
[SetUp] [DataSource("name")]
[SetUpFixture] [DeploymentItem("path")]
[Suite] [Description("text")]
[TearDown] [ExpectedException("myExcepton")]
[Test] [HostType("type")]
[TestFixture] [Ignore]
[TextFixtureSetUp] [Owner("name")]
[TextFixtureTearDown] [Priority(1)]
[TestClass]
[TestCleanup]
[TestInitialize]
[TestMethod]
[Timeout(30)]

and all assertions:

NUnit assertions TS Unit Test assertions
Assert.AreEqual Assert.AreEqual
Assert.AreNotEqual Assert.AreNotEqual
Assert.AreNotSame Assert.AreNotSame
Assert.AreSame Assert.AreSame
Assert.Contains Assert.Equals
Assert.Fail Assert.Fail
Assert.Greater Assert.Inconclusive
Assert.GreaterOrEqual Assert.IsFalse
Assert.Ignore Assert.IsInstanceOfType
Assert.IsAssignableFrom Assert.IsNotInstanceOfType
Assert.IsEmpty Assert.IsNotNull
Assert.IsFalse Assert.IsNull
Assert.IsInstanceOfType Assert.IsTrue
Assert.IsNaN
Assert.IsNotAssignableFrom
Assert.IsNotEmpty
Assert.IsNotInstanceOfType
Assert.IsNotNull
Assert.IsNull
Assert.IsTrue
Assert.Less
Assert.LessOrEqual
StringAssert.Contains
StringAssert.StartsWith
StringAssert.EndsWith
StringAssert.AreEqualIgnoringCase
StringAssert.IsMatch
CollectionAssert.AllItemsAreInstancesOfType
CollectionAssert.AllItemsAreNotNull
CollectionAssert.AllItemsAreUnique
CollectionAssert.AreEqual
CollectionAssert.AreEquivalent
CollectionAssert.AreNotEqual
CollectionAssert.AreNotEquivalent
CollectionAssert.Contains
CollectionAssert.DoesNotContain
CollectionAssert.IsSubsetOf
CollectionAssert.IsNotSubsetOf
CollectionAssert.IsEmpty
CollectionAssert.IsNotEmpty
FileAssert.AreEqual
FileAssert.AreNotEqual

I have also made a listing of all pros and cons of both unit testing frameworks.

NUnit
  • Its the de-facto standard for unit tests in .NET. 
  • Already exists for several years now and it's very stable.
  • New releases and new features on a regular basis.
  • It's very easy to extend and write your own assertions (constraint model).
  • Lots of features and assertions with support for collections and files.
  • Very good integration with all editions of Visual Studio thanks to the TestDriven.NET add-in.
  • It's open-source (yes, I see this as an advantage).
  • Has its own stand-alone test runner that can installed with a simple XCOPY deployment. 
  • It's available for both .NET 1.1, .NET 2.0 or higher.
  • Very easy to use with Selenium or WatiN
  • No out-of-the-box integration with Team Foundation Server.
Team System Unit Test
  • Out-of-the-box integration with Team Foundation Server.
  • Less users
  • Its not free and only available for the Team System editions of Visual Studio. There's also no stand-alone test runner available. When our project lead or our functional analysts like to run some integration tests, they have to install one of the Team System editions of Visual Studio in order to do so. 
  • It's has many bugs and is less stable than NUnit. One of these bugs has to do with the ExpectedException attribute. I've also experienced some unexpected behavior with Assert.AreEqual (tests that should fail but instead pass for some obscure reasons). I've also got some frequent crashes of test runs.  
  • Test runs are very slow. One of the most important characteristics of a good unit test is that it's fast.
  • Less features than other unit test frameworks like NUnit or MbUnit. Less assertions, no inheritance of test fixtures and no hierarchical organization of unit tests to name a few.
  • Only available for .NET 2.0 or higher.
  • Web testing of Team System simply sucks compared to Selenium and WatiN. It's possible to use the latter web testing frameworks in favor of the first one, but they are not well integrated.

I value the out-of-the-box integration of Team System Unit Test with TFS, but in order to sacrifice all the NUnit goodness, Microsoft has to do a better job for their unit test framework. Based on this list, information I found and my own experiences, I think NUnit is a no-brainer. I guess that after some Googling around and some tweaking, it's possible to integrate NUnit with TFS as well but I can't prove that right now (if you can help me with that, please send me an e-mail).

Therefore I would like to get your opinion on this matter. Some feedback is highly appreciated. You can also have a look at this post from Roy Osherove.

Profile picture of Jan Van Ryswyck

Jan Van Ryswyck

Thank you for visiting my blog. I’m a professional software developer since Y2K. A blogger since Y2K+5. Curator of the Awesome Talks list. Past organizer of the European Virtual ALT.NET meetings. Thinking and learning about all kinds of technologies since forever.

Comments

About

Thank you for visiting my website. I’m a professional software developer since Y2K. A blogger since Y2K+5. Curator of the Awesome Talks list. Past organizer of the European Virtual ALT.NET meetings. Thinking and learning about all kinds of technologies since forever.

Contact information

(+32) 496 38 00 82

infonull@nullprincipal-itnull.be