CI - Failed tfs team build watcher. Part II - the code.

Code follows, as promised in the first installment on this subject.

What is required to develop?

1. Resolve the "windows" name of the build requestor in CI to the Display Name and email address. This requires a call to the TFS API, as name comes in the format "domain\windowsusername".
That is done by the ResolveUser build task: http://code.google.com/p/toolsdotnet/source/browse/trunk/Tools.Net/src/Tools.TeamBuild.Tasks/ResolveUser.cs

2. Keep a build and requestor names (email address) for the first broken build following the successful one. Again I decided not to use build API for that because of its chatty character of retrieving all build records and only then filtering through them. This is achieved by the BuildGateKeeper task: http://code.google.com/p/toolsdotnet/source/browse/trunk/Tools.Net/src/Tools.TeamBuild.Tasks/BuildGateKeeper.cs

Pardon me all the surrounding files for achieving the better test coverage, you can get the dll (.exe) straight here: http://cid-c651f6a9f36fb87d.skydrive.live.com/self.aspx/Public/tools.tfs.utils.exe. Reason for having it as an exe was ability to use ResolveUser task as a console utility.

First, get those tasks imported (change path to yours):

<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\SitronicsTeamBuild.Quick\tools.tfs.utils.exe" TaskName="Tools.TeamBuild.Tasks.ResolveUser" />   

<UsingTask AssemblyFile="$(MSBuildExtensionsPath)\SitronicsTeamBuild.Quick\tools.tfs.utils.exe" TaskName="Tools.TeamBuild.Tasks.BuildGateKeeper" /> 

Then use ResolveUser in some place before the build:

<ResolveUser WindowsAccountName="$(RequestedFor)"> 

  <Output TaskParameter="DisplayName" PropertyName="Requestor" />

  <Output TaskParameter="MailAddress" PropertyName="MailAddress" />

</ResolveUser>

I used a default for our own tfs, but there is an extra parameter TfsUrl that is the best to set to the full url as http://tfsname:port.

Note that it is $(RequestedFor) that is used as $(RequestedBy) gives something like "System Build Account", while $(RequestedFor) gives user windows name. This way we are getting from mine domain\sdvoychenko to Stanislav Dvoychenko and "yours truly" email address.

If build is successful build I reward the lucky guy with the grateful email:

<CreateProperty          Value="Success" >

 <Output TaskParameter="Value" PropertyName="BuildStatus" />

</CreateProperty>

<BuildGateKeeper BuildStatus="$(BuildStatus)" StateFilePath="c:\Build\$(BuildDefinition).txt" RequestorMailAddress="$(MailAddress)" RequestorDisplayName="$(Requestor)">

</BuildGateKeeper>

<Mail SmtpServer="YourSmtpServer" IsBodyHtml="true" Username="xxx\xxx" Password="xxx" To="$(MailAddress)" CC="controlfreak@controlfreakdomain.com" From="build@controlfreakdomain.com" Body="Successful check-in! Your checkin to SP5 Dev branch was built successfuly. Check-in number and comments to be provided soon. Build log: %3CBR%2F%3E $(DropLocation)\$(BuildNumber)\BuildLog.txt;" Subject="Your checkin to SP5 Dev branch was built successfuly."></Mail>           

What BuildGateKeeper does at this moment is to delete the state file (if any), that keeps exactly who was the first to break the build.
After that, email sending follows for which you might need to add the following usage if you have not been using community tasks (http://msbuildtasks.tigris.org/) in your project already:

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />

Now, lets see what to do if build breaks.

<Copy Condition="Exists('$(MSBuildProjectDirectory)\build.errors.txt')"                SourceFiles="$(MSBuildProjectDirectory)\build.errors.txt" DestinationFolder="$(DropLocation)\$(BuildNumber)\" />

<ReadLinesFromFile Condition="Exists('$(DropLocation)\$(BuildNumber)\build.errors.txt')"            File="$(DropLocation)\$(BuildNumber)\build.errors.txt" >

  <Output TaskParameter="Lines"  ItemName="BuildErrorLines"/>

</ReadLinesFromFile>

<CreateProperty Value="Failure" >

  <Output TaskParameter="Value" PropertyName="BuildStatus" />

</CreateProperty>

<BuildGateKeeper BuildStatus="$(BuildStatus)" StateFilePath="c:\Build\$(BuildDefinition).txt" RequestorMailAddress="$(MailAddress)" RequestorDisplayName="$(Requestor)">

  <Output TaskParameter="BreakerMailAddress" PropertyName="BreakerMailAddress" />

  <Output TaskParameter="BreakerDisplayName" PropertyName="BreakerDisplayName" />

  <Output TaskParameter="BreakTimeStamp" PropertyName="BreakTimeStamp" />

</BuildGateKeeper>

<Mail SmtpServer="YourSmtpServer" IsBodyHtml="true" Username="xxx\xxx" Password="xxx" To="$(MailAddress)" CC="controlfreak@controlfreakdomain.com;$(BreakerMailAddress)" From="build@controlfreakdomain.com" Body="You got this email message because the code you checked in recently was not compiled successfully during verification build. Changeset number and comments to be provided soon. &lt;br/&gt; The build was broken first by $(BreakerDisplayName) at $(BreakTimeStamp). &lt;br/&gt;&lt;br/&gt; Errors log: &lt;br/&gt; &lt;b&gt; @(BuildErrorLines) &lt;/b&gt; &lt;br/&gt; &lt;br/&gt; More detailed info: &lt;a href='file:///$(DropLocation)\$(BuildNumber)\BuildLog.txt' &gt; $(DropLocation)\$(BuildNumber)\BuildLog.txt &lt;/a &gt;. &lt;br/&gt; If you believe this informatin is false positive or need help with fixing check-in, please reply to the Dvoychenko Stanislav." Subject="FAILURE! Build - $(BuildNumber) - changes your checked in were not compiled successfuly."></Mail>

The line I owe here is an update to the .rsp build file in the BuildTypes folder. That is to log build errors only to the separate file. I'll add this snippet asap.
What we do then is to copy that error file to the drop location and then we read its lines into the $(BuildErrorLines) property which we use then in the body of our email.

We then use BuildGateKeeper to create a break state file if there is no such yet. If there is already one, then task uses it to populate Breaker details which we use again in our Mail community task to send the email to the check-in initiator and CC to the guy who First Broke The Build (and the control freak who is me by the way:).

This is very spartan and I'll share the improvements as I progress in the evenings :).

Happy hunting my fellow control freaks!

P.S. Full solution including nunit tests is available here:  http://code.google.com/p/toolsdotnet/source/browse/trunk/Tools.Net/Tools.Build.sln

CI - Failed team build watcher. Part I - main ideas.

This is my DIY activity, so sorry if something like this already exists, I have not been searching that long. I'd highly recommend to look at the buddy build http://www.codeplex.com/BuddyBuild as well as gated check-ins feature in Team System 2010 http://blogs.msdn.com/jimlamb/archive/2009/01/23/coming-soon-gated-check-in.aspx will make this quite obsolete.

Problem definition:

1. First Developer check-ins files, breaking the build
Action required: Send notification email with build errors list and link to the detailed log file.

2. Second Developer check-ins files, his check-in doesn't build as build is broken by the First Developer
Action required: Send notification email to the Second Developer saying build can't be verified. CC the First Developer on it (knock-knock on his conscious).

3. Third Developer check-ins -> same as the second.

4. Corrective check-ins -> team build builds
Action required: Notify the check-in author of successful check-in

*Names of all devs in the sreenshots are mine as an implication of testing on myself. As well as error message is faked in the team build target so I don't need to wait for the compilation to complete.

Implementation ideas:

1. Don't use the Team Build API as its query ability is quite limited. In this case we need the first build breaker after the last successful build.
2. Even if possible to code completely with bunch of msbuild and community build tasks http://msbuildtasks.tigris.org/, still I decided to go with custom task(s), except for sending email I used Email community task and for reading error logs I used msbuild ReadLines task.
Main reason for this decision was ability to provide unit/integration tests instead of taming/tuning a bunch of xml (Still there is enough xml left :))
3. Introduce failure state file to keep the data of the build breaker immediately following the successful build. That is to compensate the absence of the "build query language" already referred to in point #1.

Implementation time required - 5 hours
Test coverage achieved at the moment - 60%
Lines of code - to be counted
Implementation guide, source and binaries - coming in the next post

Hands-on - Building msbuild projects in parallel

I'm currently looking into optimizing our team builds from their average 50 minutes to the under 10 minutes.

Key points of course will be incremental build and get, which we have a twist with because of code-generated ORM in one place - will be solving later this week...

But for this evening exercise I was looking into the multi processor msbuild.
Starting points were following entries at the msbuild team blog:
http://blogs.msdn.com/msbuild/archive/2007/10/22/enabling-multiprocessor-support-in-an-msbuild-host.aspx
http://blogs.msdn.com/msbuild/archive/2007/04/26/building-projects-in-parallel.aspx

My lessons from the above links are:
- We still have to use /m switch of msbuild to build projects in parallel, just BuildInParallel MSBuild task attribute is not enough.
- A special treatment should be given for cases of sequential dependencies, when one project depends on another (but it should be regardless the parallel factor ...).

To play with parallel build I created a spike project. You can find it attached to this post or at the following svn url:

The project is a combination of the above mentioned posts at the msbuild blog with few twists:
- To avoid dependency on the "sleep" ms-dos utility I added a simple SleepTask directly to the hands-on project.
- I combined the projects structure of the first post with dependency resolution per the second link.

Project structure looks like:

 

With 1.proj being:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

  <Target Name="t">

    <Message Importance="high" Text="## starting 1 ##"/>

    <SleepTask Time="3000" />

    <Message Importance="high" Text="## finishing 1 ##"/>

  </Target>

</Project>
 

2.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

  <Target Name="t">

    <Message Importance="high" Text="## starting 2 ##"/>

    <SleepTask Time="3000" />

    <Message Importance="high" Text="## finishing 2 ##"/>

  </Target>

</Project>

3.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

    <ItemGroup>

      <ProjectReference Include="2.proj"/>

    </ItemGroup>

    <Target Name="t" DependsOnTargets="BuildProjectReferences">

      <Message Importance="high" Text="## starting 3 ##"/>

      <SleepTask Time="3000" />

      <Message Importance="high" Text="## finishing 3 ##"/>

    </Target>

    <Target Name="BuildProjectReferences">

      <MSBuild Projects="@(ProjectReference)" />

    </Target>

</Project>

Than we have two flavors of "root" project, one is running 1.proj and 2.proj and we kick it off by msbuild build12.proj /m:

Whereas build123.proj builds 1,2,3 in parallel. You can see from the output of msbuild /m, that project was built as expected, taking a bit more than 6 seconds and running 2.proj in advance of 3.proj (and only once:))

Midnight approaching and this is it for today. For completeness, providing the source for remaining projects and a SleepTask:

build12.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <Target Name="t">

    <Message Importance="high" Text="## in root building children ##"/>

    <MSBuild Projects="1.proj;2.proj;" BuildInParallel="true"/>

    <Message Importance="high" Text="## in root done building ##"/>

  </Target>

</Project>

build123.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <Target Name="t">

    <Message Importance="high" Text="## in root building children ##"/>

    <MSBuild Projects="1.proj;2.proj;3.proj" BuildInParallel="true"/>

    <Message Importance="high" Text="## in root done building ##"/>

  </Target>

</Project>

SleepTask:

using Microsoft.Build.Utilities;

using Microsoft.Build.Framework;

using System;

using System.Threading;

using System.Globalization;

namespace MultiProcBuild

{

    /// <summary>

    /// MSBuild task to delay the process for the required amount of time.

    /// </summary>

    /// <remarks>Realized via call to Thread.Sleep.</remarks>

    public class SleepTask : Microsoft.Build.Utilities.Task

    {

        /// <summary>

        /// Amount of time in milliseconds to sleep.

        /// </summary>

        [Microsoft.Build.Framework.Required()]

        public int Time { get; set; }

        public override bool Execute()

        {

            try

            {

                Thread.Sleep(Time);

                Log.LogMessage(MessageImportance.High, String.Format(CultureInfo.InvariantCulture,

                    "Task {0}, slept for {1} ms.", GetType().Name, Time.ToString()));

                return true;

            }

            catch (Exception ex)

            {

                Log.LogErrorFromException(ex, true);

                return false;

            }

        }

    }

}

Project url: http://code.google.com/p/toolsdotnet/source/browse/#svn/trunk/Tools.Net/spikes/Versioning/MultiProcBuild
Zipped sources: http://cid-c651f6a9f36fb87d.skydrive.live.com/self.aspx/Public/MultiProcBuild.zip