What drives Fabrikam’s development process is the desire to quickly satisfy their customer’s changing demands. As the more agile of the two companies in our story, Fabrikam’s online users might see updates every couple of weeks or even days. Even in their more conventional projects, clients are invited to view new features at frequent intervals.

These rapid cycles please the customers, who like to see steady progress and bugs fixed quickly. They also greatly reduce the chances of delivering a product that doesn’t quite meet the client’s needs. At the same time, they improve the quality of the product.

At Contoso, the more traditional of the two, employees are suspicious of Fabrikam’s rapid cycles. They like to deliver a high quality product, and testing is taken very seriously. Most of their testing is done manually, exercising each user function by following scripts: click this button, enter text here, verify the display there. It can’t be repeated every night. They find it difficult to see how Fabrikam can release updates so rapidly and yet test properly.

But they can, thanks to automation. Fabrikam creates lots of tests that are written in program code. They test as early as possible, on the development machine, while the code is being developed. They test not only the features that the user can see from the outside, but also the individual methods and classes inside the application. In other words, they do unit testing.

Unit tests are very effective against regression—that is, functions that used to work but have been disturbed by some faulty update. Fabrikam’s style of incremental development means that any piece of application code is likely to be revisited quite often. This carries the risk of regression, which is the principal reason that unit tests are so popular in Fabrikam and agile shops like them.

This is not to deny the value of manual testing. Manual tests are the most effective way to find new bugs that are not regressions. But the message that Fabrikam would convey to their Contoso colleagues is that by increasing the proportion of coded tests, you can cycle more rapidly and serve your customers better.

For this reason, although we have lots of good things to say about manual testing with Visual Studio, we’re going to begin by looking at unit testing. Unit tests are the most straightforward type of automated test; you can run them standalone on your desktop by using Visual Studio.

Tip: Unit testing does involve coding. If your own speciality is testing without writing code, you might feel inclined to skim the rest of this chapter and move rapidly on to the next. But please stick with us for a while, because this material will help you become more adept at what you do.

The topics for this blog are:

• Unit testing on the development machines.
• Checking the application code and tests into the source code store.
• The build service, which performs build verification tests to make sure that the integrated code in the store compiles and runs, and also produces installer files that can be used for system tests.

Prerequisites

To run unit tests, you need Visual Studio on your desktop.

The conventions by which you write and run unit tests are determined by the unit testing framework that you use. MSTest comes with Visual Studio, and so our examples use that. But if you already use another unit testing framework such as NUnit, Visual Studio 2012 will recognize and run those tests just as easily, through a uniform user interface. It will even run them alongside tests written for MSTest and other frameworks. (Visual Studio 2010 needs an add-in for frameworks other than MSTest, and the integration isn’t as good as in Visual Studio 2012.)

Later in this blog, we’ll talk about checking tests and code into the source repository, and having the tests run on your project’s build service. For that you need access to a team project in Visual Studio Team Foundation Server.

Unit tests in Visual Studio

A unit test is a method that invokes methods in the system under test and verifies the results. A unit test is usually written by a developer, who ideally writes the test either shortly before or not long after the code under test is written.

To create an MSTest unit test in Visual Studio, pull down the Test menu, choose New Test, and follow the wizard. This creates a test project (unless you already had one) and some skeleton test code. You can then edit the code to add tests:

C#
[TestClass]
public class RooterTests
{
[TestMethod] // This attribute identifies the method as a unit test.
public void SignatureTest()
{
// Arrange: Create an instance to test:
var rooter = new Rooter();
// Act: Run the method under test:
double result = rooter.SquareRoot(0.0);
// Assert: Verify the result:
Assert.AreEqual(0.0, result);
}
}

Each test is represented by one test method. You can add as many test methods and classes as you like, and call them what you like. Each method that has a [TestMethod] attribute will be called by the unit test framework. You can of course include other methods that are called by test methods. If a unit test finds a failure, it throws an exception that is logged by the unit test framework.

Running unit tests

You can run unit tests directly from Visual Studio, and they will (by default) run on your desktop computer. (More information can be found in the MSDN topic Running Unit Tests with Unit Test Explorer.) Press CTRL+R, A to build the solution and run the unit tests. The results are displayed in the Test Explorer window in Visual Studio.

(The user interface is different in Visual Studio 2010, but the principles are the same.) If a test fails, you can click the test result to see more detail. You can also run it again in debug mode. The objective is to get all the tests to pass so that they all show green check marks. When you have finished your changes, you check in both the application code and the unit tests. This means that everyone gets a copy of all the unit tests. Whenever you work on the application code, you run the relevant unit tests, whether they were written by you or your colleagues. The checked-in unit tests are also run on a regular basis by the build verification service. If any test should fail, it raises the alarm by sending emails.

Debugging unit tests

When you use Run All, the tests run without the debugger. This is preferable because the tests run more quickly that way, and you don’t want passing tests to slow you down. However, when a test fails, you might choose Debug Selected Tests. Don’t forget that the tests might run in any order.

Test-first development

Writing unit tests before you write the code—test-first development—is recommended by most developers who have seriously tried it. Writing the tests for a method or interface makes you think through exactly what you want it to do. It also helps you discuss with your colleagues what is required of this particular unit. Think of it as discussing samples of how your code will be used. For example, Mort, a developer at Fabrikam, has taken on the task of writing a method deep in the implementation of an ice-cream vending website. This method is a utility, likely to be called from several other parts of the application. Julia and Lars will be writing some of those other components. They don’t care very much how Mort’s method works, so long as it produces the right results in a reasonable time.

Mort wants to make sure that he understands how people want to use his component, so he writes a little example and circulates it for comment. He reasons that although they aren’t interested in his code, they do want to know how to call it. The example takes the form of a test method. His idea is that the test forms a precise way to discuss exactly what’s needed.

Tip: Think of unit tests as examples of how the method you’re about to write will be used.

Mort frequently writes tests before he writes a piece of code, even if his own code is the only user. He finds it helps him get clear in his mind what is needed.

Limitations of test-first development?

Test-first development is very effective in a wide variety of cases, particularly APIs and workflow elements where there’s a clear input and output. But it can feel less practical in other cases; for example, to check the exact text of error reports. Are you really going to write an assertion like:

C#
Assert.AreEqual(“Error 1234: Illegal flavor selected.”, errorMessage);

Baseline tests

A common strategy in this situation is to use the baseline test. For a baseline test, you write a test that logs the output of your application to a file. After the first run, you verify manually to see that it looks right. For subsequent runs, you write test code that compares the new output to the old log, and fails if anything has changed. It sounds straightforward, but typically you have to write a filter that allows for changes like time of day and so on.

Many times, a failure occurs just because of some innocuous change, and you get used to looking over the output, deciding there’s no problem, and resetting the baseline file. Then on the sixth time it happens, you miss the crucial thing that’s actually a bug; and from that point onwards, you have a buggy baseline. Use baseline tests with caution.

Tests verify facts about the application, not exact results

Keep in mind that a test doesn’t have to verify the exact value of a result. Ask yourself what facts you know about the result. Write down these facts in the form of a test. For example, let’s say we’re developing an encryption method. It’s difficult to say exactly what the encrypted form of any message would be, so we can’t in practice write a test like this:

C#

string actualEncryption = Encrypt(“hello”);
string expectedEncryption = “JWOXV”;
Assert.AreEqual(expectedEncryption, actualEncryption);

But wait. Here comes a tip:

Tip: Think of facts you know about the result you want to achieve. Write these as tests.

What we can do is verify a number of separate required properties of the result, such as:

C#
// Encryption followed by decryption should return the original:
Assert.AreEqual (plaintext, Decrypt(Encrypt(plaintext)));
// In this cipher, the encrypted text is as long as the original:
Assert.AreEqual (plaintext.Length, Encrypt(plaintext).Length);
// In this cipher, no character is encrypted to itself:
for(int i = 0; i < plaintext.Length; i++)
Assert.AreNotEqual(plaintext[i], Encrypt(plaintext)[i]);

Using assertions like these, you can write tests first after all. How to use unit tests In addition to test-first (or at least, test-soon) development, our recommendations are:

• A development task isn’t complete until all the unit tests pass.
• Expect to spend about the same amount of time writing unit tests as writing the code. The effort is repaid with much more stable code, with fewer bugs and less rework.
• A unit test represents a requirement on the unit you’re testing. (We don’t mean a requirement on the application as a whole here, just a requirement on this unit, which might be anything from an individual method to a substantial subsystem.)
• Separate these requirements into individual clauses. For example:
• Return value multiplied by itself must equal input AND
• Must throw an exception if input is negative AND ….
• Write a separate unit test for each clause, like the separate tests that Mort wrote in the story. That way, your set of tests is much more flexible, and easier to change when the requirements change.
• Work on the code in such a way as to satisfy a small number of these separate requirements at a time.
• Don’t change or delete a unit test unless the corresponding requirement changes, or you find that the test does not correctly represent the intended requirement.