In-Depth

Levels of TFS Build Automation

Your team won't achieve automated build nirvana overnight. Here's how to break down the process and adopt the changes in the Microsoft build system.

• By Benjamin Day
• 07/01/2011

One of the biggest areas of change in Microsoft Team Foundation Server (TFS) 2010 from its predecessor TFS 2008 is the improved build system. It was updated with a new build script format and server architecture driven by Windows Workflow (XAML). The result is that builds are now easier to set up and customize. And the new build framework helps you avoid the problem of broken builds altogether.

If you're starting from "no automated builds," it's still going to take a while. For a lot of development shops, adopting build automation can be overwhelming, and teams need a way to break it into a gradual process with varying levels of sophistication. This article will walk you through the adoption process and discuss the various levels of TFS build automation from simply compiling on a build server and running unit tests to automating continuous integration and integrating your builds with Lab Management and coded UI tests.

You might be asking yourself why teams would use a separate build server and add this additional complexity to their lives and development process. The short answer is: to try to eliminate the "works on my box" problem.

The build server is the neutral third-party that verifies that your code actually compiles. It helps automate a lot of the drudgery of your build, test and release process. The goal is to make handing off something to Quality Assurance (QA) testers or deploying to production a non-event.

Use of a build server also helps prove that you know how to deploy your application and that you and your team have dealt with all the little issues. If you can't tell the build server how to compile and package your app, you probably don't actually know how to do it.

Monthly Builds: That Was Then
Thinking back to the days before TFS Build or CruiseControl, I'd frequently see customers creating a build only about once per month.

Sure, the developers were doing "Get Latest" on the source repository at least once per day and everyone's code basically worked, but when they tried to package it up to ship it or "kick it over the wall to QA," it tended to be an ordeal: All hands on deck, don't make any plans for tonight, we're ordering pizza, we're doing a build.

Why did this always happen on the build servers when the app "basically worked" on the developers' machines? Think about what happens whenever a new developer joins the team. How long does it take to get that developer's workstation configured so that the app will compile and actually run? There are so many tweaks and other things that have to happen to the local workstation.

"Oh ... you forgot to install the third-party library for XYZ. And don't forget to set the registry keys for [fill in the blank]. That error's happening because you didn't copy the latest version of the configuration file to the foo directory. And don't forget to change the connection string to your local database. And make sure you get the right version of the database deploy script because remember we made that change to the security stored procedure two weeks ago."

Once you get all that stuff done, you're cruising again -- but it's a bit of a productivity killer. And, of course, you documented that whole process so that it's not nearly as painful next time, right?

It used to be so painful on the build server when the code basically worked on developers' workstations because, first, you were probably tearing down the build environment each time to eliminate any leftover stuff from last month's build. Second, the dev workstations were gradually patched every day with every little configuration change, but the build server didn't get used very often. This meant that a month's worth of little tweaks got lumped into a single day. Your team probably had to debug each and every one of these little changes because you basically forgot everything you did -- you had to remember what not to forget and it didn't take long before your brain "garbage collected" those details.

The automated build is going to force you to automate this whole build process so that a computer can do it. Once you do that, you can run builds all the time.

Levels of Build Awesomeness
As a consultant who works with customers to nail down and improve their software development process, I tend to want to recommend that organizations use everything in the TFS/Visual Studio application lifecycle management (ALM) stack. I can say: "You should build everything, deploy the database, populate the database with test data, spin up a virtual environment with TFS Lab Management, deploy the binaries, set all the connection strings for all possible environments, run the unit tests, run all the coded UI tests and run load tests for every check-in using TFS Build." But if you're starting from "no automated builds," it's going to take a while. It's a fair amount of work and you probably won't achieve automated build nirvana overnight.

For a lot of developers and development shops, looking at the mountain of opportunity can be overwhelming, and you may not know where to start. You need a way to break it apart into something you can adopt gradually without bringing your entire development team to a dead stop while trying to get everything perfect. That brings us to what I call the "levels of build awesomeness":

1. "Get Latest" deploy on the developer desktop machine
2. Compile on a build server
3. Run unit tests on the build server
4. Continuous Integration (CI)
5. Gated Check-in
6. Deploy to QA/test
7. Lab Management with coded UI tests

First off, you need to solve any problems on your developer workstations. This is where all the important stuff happens, and it's the first line of defense against productivity problems. You should be able to take a brand-new developer machine, connect to TFS, right-click the solution root of your application, do a "Get Latest" and then immediately run "Build Solution" and have the application run. Granted, you might have to install some third-party components first and get the database configured, but when you pull that code down from TFS for the first time, it should just work.

If you can get this working properly, it means that you understand what it takes to build the app. It eliminates a lot of developer productivity headaches -- and makes building on a TFS Build server fairly straightforward. If you're developing an ASP.NET Web application, you should convert your Web projects to use IIS instead of the development Web server. This helps you to ensure that the developer environment is the same as the build environment, which needs to match the production environment as closely as possible. It has the added benefit of forcing you to figure out the IIS configuration as well as the IIS App Pool configuration details.

Compile on a Build Server
When you've figured out the details of how to deploy and build your solution on your developer desktop, you're ready to move on to the next level of build sophistication and compile your code on a build server. If all you want to do is compile your code, this process is surprisingly simple.

TFS build definitions are contained inside of a Team Project. You're going to start by navigating to the Team Project that will contain your build. Usually, this is the same Team Project that contains your source code. From the Team Project, navigate to the Builds node, right-click and select "New Build Definition" (see Figure 1).

[Click on image for larger view.]

Figure 1. Create a new build definition.

You should now see the build definition editor, which allows you to modify the name and description for the build. This includes information about how and when the build should be started, what the source control workspace definition is, the Build Controller that should run the build, how long the build outputs (for instance, the compiled output of the build) should be retained, and details for the build script execution.

The core configuration options are on the Process tab of the build editor dialog, as shown in Figure 2. Whereas the other tabs in the build editor dialog box control the metadata about the build, the Process tab provides the options for what happens during the actual build execution. You can choose which build script to use (by default, it uses the build script defined in DefaultTemplate.xaml), which Solutions (*.sln) to build, whether to run automated tests, the build number naming convention, build log output verbosity and static code analysis options.

[Click on image for larger view.]

Figure 2. The build configuration options are found on the Process tab of the build editor.

Advanced options allow you to disable and skip steps within the default build script template.

After you populate all the required fields on your new build definition and click Save, the build definition will appear in Team Explorer in the Builds node of your Team Project.

From here, you can right-click the build definition and choose Queue New Build to start the build. After you've started the build, the Build Explorer window will pop up and you should see your build running in the Queued tab (see Figure 3).

[Click on image for larger view.]

Figure 3. After you've started the build, the Build Explorer window will pop up and you should see your build running in the Queued tab.

From here you can double-click on the running build and bring up the real-time details to see what the build is doing and whether it's passed, as shown in Figure 4. Ideally, your build passes -- but when it doesn't, the build details screen helps you figure out why a build has failed, whether it's compilation, unit test execution failure, something related to your build server or, if you've customized it, the build script itself.

[Click on image for larger view.]

Figure 4. Double-click on the running build in Build Explorer to bring up the real-time execution details to see what the build is doing and whether it passed.

At this point you have the code compiling on someone else's box and, even if you don't do anything else, this is helpful. You can really drive quality and increase the confidence that the code is working if you run unit tests. Assuming that the code compiled successfully, the default build process settings will trigger the execution of unit tests in any DLL that contains the word "test." In a perfect world, your unit tests would always pass as part of your automated build because that helps you eliminate the "works on my box but doesn't work on anyone else's box" problem. That compilation step establishes the first level of quality. The unit tests get you to the next level of quality where you know that the unit tests work in multiple places.

Continuous Integration
Up to this point, these builds have all been started manually: someone had to actually go to Team Explorer and start the build. You've managed to clear the hurdle of scripting out the basics of the build so that it's no longer a special event but it still involves a conscious, human decision to kick it off. This is where the next levels of build awesomeness come in: builds that are triggered by a check-in.

Let's think this through a little bit. Assume for a moment that you're manually triggering your build on a fairly regular basis. You're probably spending a decent amount of time fixing little random compilation errors, and you're almost definitely running into broken unit tests. If you value your unit tests passing -- and you should -- you really want to make sure that they're passing all the time.