Building in Parallel Across Multiple Build Agents in TFS2012 for Metro Apps

By jay at March 30, 2013 12:50 Tags: , , , , , , ,

TL;DR: Using an upgraded (and fixed) Parallel Build Process Template allows to use multiple TFS2012 build agents simultaneously, which can be more than welcome when building metro apps that target all three supported platforms. A build that took 11 minutes can go down to 3.5 minutes.

Download the Parallel Build Process Template for TFS2012 here.


CI is a wonderful feature, especially when associated with Gated Checkins.

You’re certain that what’s in your source control is in line with your build definition and constraints, and that there is always a binary that respects a minimum set of rules. This does not ensure that your app is bug free, but still, that’s a minimum.


Build time matters

The downside of this validation is that there cannot be multiple builds running at the same time. This can become a bottleneck when multiple developers checkin within the duration of a single build run.

This means that the longer your build gets, the longer a developer might wait for its task completion because of a long build queue, and increase its task switching cost. If a build fails, the developer needs to unshelve its changes, make the necessary adjustments, then check-in again.

Below 4 minutes of build time, this stays in the acceptable range where the developer’s task context may not be lost if the build fails.


Configuring Multiple TFS 2012 Build Services on one Machine

By jay at October 27, 2012 12:18 Tags: , , ,

Since TFS2010, because there’s been the introduction of project collections, build agents are quite the trouble for IT admins.

On one hand, there’s the fact that putting every project in a single collection can be a maintenance and project isolation nightmare, and on the other hand the fact that there can officially be one build controller per machine tied to a single project collection, forcing to have many machines to support continuous integration build environments.

Luckily, there’s been a workaround published two years ago on how to run multiple build services on the same machine, and it’s working pretty fine.


However with TFS2012, this hack needs to be adjusted to work properly.

Prior to running the configuration tool, instead of setting this:

set TFSBUILDSERVICEHOST=buildMachine-collection2

The new environment variable is:

set TFSBUILDSERVICEHOST.2012=buildMachine-collection2

I’m guessing that this can be used to run both 2010 and 2012 build services on the same machine without interferences, though I haven’t tried it.

Just as a reminder, to use this feature, you need to set non-overlapping build working folders and different listening ports for every new instance. This will avoid conflicts…

Works pretty great, now with TFS2012 !

NuGet package customizations and optional references

By jay at November 25, 2011 20:55 Tags: , , , ,

This article describes a bit of what NuGet does and why you should take a look at it, but also a package installation customization to a work around a problem with packages that contain optional assemblies.


NuGet is a fantastic tool. Its ability to ease package discovery, installation and update is a big time saver. As a solution maintainer, you can spend less time deploying and updating your external dependencies, particularly when they’re often updated.


Private NuGet Repositories

It can be used to easily install public packages exposed via Microsoft’s servers, but it can also be used to create private package repositories.

I've been using it to publish internal framework that is often updated and used in many projects. The automatic creation of packages in a build script and the ease of deployment using the NuGet OData server is, again, a big time saver.

Each time a check-in is performed to update the internal framework, a new package is automatically created and it appears as a package update for the projects that use it.


Existing libraries and NuGet

The package model is built around some basic rules:

  • Each package is installed using its version as the directory name
    The NuGet engine will update all the references “HintPath” nodes of projects files for them to point to the updated package location.
  • By default, the install action of a package adds references to all the available assemblies in the target project.
    There’s a pretty good reason for that; you don’t want to download and install assemblies that you don’t need. There is no support for “sub-packages”, even with reference exclusions. Big frameworks, like Enterprise Library, get chunked into small dependent packages, and you can install only those needed by your projects.
  • A project that installed a package that has references to package-excluded references will not have these manual references updated to the latest package.
    That is a side effect of the second rule. If you add reference-excluded assemblies to a package, the action of updating that package will not update manual references you created by referencing these assemblies.


    For existing libraries that may not easily be split into small pieces, primarily for time constraints, moving to NuGet can be a tough job. If you make a package with all your assemblies and only reference a few of them by default in there, then updating the package can quickly become an annoying “Find and Replace” job to manually change references that did not get updated automatically by the NuGet engine.


    Using the Install.ps1 script

    Fortunately, there is a file that can be included in the package, which is by convention named and located in tools\install.ps1.

    It is a PowerShell script that gets called automatically when the package is installed. But the interesting part is that this installer file gets called with a DTE.Project instance that can be used to manipulate the actual project in VS2010 !

    This means that when the package is installed, it is possible to update the references that were manually created to the previous package assemblies.


    Updating the manual references

    This is not a straightforward as it may seem, but after working around a HintPath update issue, here is a small helper to do the job:

    param($installPath, $toolsPath, $package, $project)
        $packageName = $package.Id + '.' + $package.Version;
        $packageId = $package.Id;
        # Full assembly name is required
        Add-Type -AssemblyName 'Microsoft.Build, Version=, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
        $projectCollection = [Microsoft.Build.Evaluation.ProjectCollection]::GlobalProjectCollection
        # There is no indexer on ICollection<T> and we cannot call
        # Enumerable.First<T> because Powershell does not support it easily and
        # we do not want to end up MethodInfo.MakeGenericMethod.
        $allProjects = $projectCollection.GetLoadedProjects($project.Object.Project.FullName).GetEnumerator(); 
            foreach($item in $allProjects.Current.GetItems('Reference'))
                $hintPath = $item.GetMetadataValue("HintPath")
                $newHintPath = $hintPath -replace $packageId + ".*?\\", "$packageName\\"
                if ($hintPath -ne $newHintPath)
                    Write-Host "Updating $hintPath to $newHintPath"
                    $item.SetMetadataValue("HintPath", $newHintPath);

    This script is called for each project the package is installed onto, and scans all the references of the project that match the current package to update them.

    You’ll notice that the ICollection<T> interface is not particularly PowerShell friendly. Too bad the Powershell syntax does not allow the use of generic methods, otherwise that nasty GetEnumerator / MoveNext could have gone away. Still, Powershell is dynamically typed so using IEnumerable.Current is fine.

Team Build and Windows Phone 7

By jay at May 01, 2011 00:00 Tags: , , ,

Building Windows Phone 7 applications in an agile way encourages the use of Continuous Integration, and that can be done using Team System 2010.

There are a few pitfalls to avoid to get there, but this can be acheived quite easily with great results.

I won't cover the goodness of automated builds, this has already been covered a lot.


Adding Unit Tests

Along with the continous integration to create hopefully successful builds out of every check-in of source code, you'll also find the automated execution of unit tests. Team System has the ability to provide nice code coverage and unit tests success rates in Reporting Services reports, where statistics can be viewed, which gives good health indicators of the project.

Unfortunately for us, at this point there are no ways to automatically execute tests using the WP7 .NET runtime. But if you successfuly use an MVVM approach, your view models and non UI code can be compiled for multiple platforms, because they do not rely on the UI components that may be specific to the WP7 platform. That way, we are still able to test our code using the .NET 4.0 runtime with MSTest and Visual Studio 2010 test projects.

To avoid repeating code, multi-targeted files can be integrated into single target projects either by :

  • Using the Project Linker tool and link multiple projects,
  • Creating project files in the same folder and use the "include file" command when showing all files in the solution explorer. Make sure to change the output assembly name to something like "MyAssembly.Phone.dll" to avoid conflicts.

Multi-targeted files are using the #if directive and the WINDOWS_PHONE define, or the lack thereof, to compile code for the current runtime target.

There is also the option of creating projects with the Portable Library add-in, but there are some caveats on that side, and there are a few constraints when using this method. You may need to externalize code that is not supported, like UrlDecode.

Testing with MSTest ensures that your code runs successfully on .NET 4.0 runtime, but this does not test on the WP7 runtime. So to be sure that your code is successfully running on it, and since Windows Phone 7 is build on Silverlight 3, tools like the SL3 Unit Test framework can be used to manually run tests in the emulator. This cannot be integrated into the build for now, unfortunately; you'll have to place that in your QA tests.


TeamBuild with the WP7 toolkit

To be able to buid WP7 applications on your build machine, you need to install the SDK on your build machine, and a Team Build agent and/or controller.

Creating a build definition is done the same way as for any other build definition, except for one detail. You need to set the MSBuild platform to x86 instead of Auto if your build machine is running on a 64 Bits Windows. This forces the MSBuild runtime to use the 32 bits runtime, and not 64 bits, where the SDK does not work properly.

If you don't, when building your WP7 application, you'll find that intriguing message :

Could not load file or assembly 'System.Windows, Version='

Which is particularly odd considering that you've already installed the SDK, and that dll is definitely available.

You may also find that if you install that DLL from the SDK in the GAC, you'll get that other nice message :

Common Language Runtime detected an invalid program.

Which is most commonly found when mixing 32 bits and 64 bits assemblies, for which the architecture has been explicitly specified instead of "Any CPU". So don't install that DLL in the GAC and set the MSBuild architecture to x86.


That's it  for now, and Happy WP7 building !

A bit of IT in developer's world: services.exe high CPU usage

By jay at March 24, 2011 12:36 Tags: , , ,

One of the advantages of virtualization is the P2V (Physical to Virtual) process: Converting an "old" build machine to a VM so it can be moved around with the load as-is, snapshotted, backed-up and so on.

This is particularly useful when say, you have a build machine that's been there for a very long time, has a lot of dependencies over old third party software, has been customized by so many people (that have long left the company) that if you wanted to rebuild that machine from scratch, it would literally take you weeks of tweaking to get it to work properly. And that machine is running out of very old hardware that may break at any time. And that the edition of Windows that does not migrate easily to new hardware because of HAL or Mass Storage issues, requiring a reinstallation. That a lot of "ands".

That's the kind of choice you do not need to make: You just take the machine and virtualize it using SCVMM 2008 R2.

But still, even virtualized, the machine been there that long, and things have started falling apart, like having the services.exe process taking 100% of the CPU. And I did not want to have to rebuild that machine just because of that strange behavior.

If you read scott hanselman's blog, you've been recalled that Windows Server 2008 and later has the resource monitor that gives a wealth of information about the services running under services.exe. But if you're out of luck, like running under Windows Server 2003, you can still use Process Explorer. This will give you the similar kind of insight in the Windows Services that are running.

For my particular issue, this was actually the Event Log service that was taking all the resources.


How about I get my CPU back ?

After some digging around, I noticed that :

  • All Event Viewer logging sections in the MMC snap-in were all displaying the same thing, which was actually a mix of all the System, Application and Security logs.
  • Displaying any of these logs was taking a huge amount of time to display.
  • The HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\Source was containing something like "System System System System System System System" a hundred of times, the same thing for the keys for the other event logs
  • A whole bunches (thousands) of interesting sources named like some .NET application domain created by the application being built on this machine

To fix it, a few steps in that order :

  • Disable the Event Log service and reboot. You won't be able to stop it, but at the next reboot it will not start.
  • In the C:\WINDOWS\system32\config folder, move the files *.evt to a temporary folder, so they don't get picked up by the service when it'll restart
  • In the registry, for each HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application]\Source, replace the content with the one found on the same key on another very similar Windows Server 2003 machine. You can install a brand new machine and pick up the content.
  • If you have, like I did, a whole bunch of sources that look familiar and should not be there under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application], remove their keys if you removed them from the "Source" value.
  • Set the Event Log service to "Automatic" and reboot.


The interesting part about the virtualization of that build machine is like in many other occasions, the snapshots, where you can make destructive changes and go back if they were actually too destructive.


What's with the event log "interesting sources" ?

The application being built and tested is running tests on the build machine, and it makes use of application domains and log4net. Log4net has an EventLogAppender that allows the push of specific content to the Windows Event Log. Log4net defaults the name of the source to the application domain name, if there is no entry assembly.

Those tests were actually using a default configuration, and were logging Critical messages to the event log, but the domains were created using a new GUID to avoid supposed name collisions. This is something that did actually more harm than good in the long run, because each new appdomain that was logging to the event log was creating a new event source.

And the build system has been there for a long time. Hence the thousands of "oddly named" event sources.


About me

My name is Jerome Laban, I am a Software Architect, C# MVP and .NET enthustiast from Montréal, QC. You will find my blog on this site, where I'm adding my thoughts on current events, or the things I'm working on, such as the Remote Control for Windows Phone.