Hands-on - Building msbuild projects in parallel

I'm currently looking into optimizing our team builds from their average 50 minutes to the under 10 minutes.

Key points of course will be incremental build and get, which we have a twist with because of code-generated ORM in one place - will be solving later this week...

But for this evening exercise I was looking into the multi processor msbuild.
Starting points were following entries at the msbuild team blog:
http://blogs.msdn.com/msbuild/archive/2007/10/22/enabling-multiprocessor-support-in-an-msbuild-host.aspx
http://blogs.msdn.com/msbuild/archive/2007/04/26/building-projects-in-parallel.aspx

My lessons from the above links are:
- We still have to use /m switch of msbuild to build projects in parallel, just BuildInParallel MSBuild task attribute is not enough.
- A special treatment should be given for cases of sequential dependencies, when one project depends on another (but it should be regardless the parallel factor ...).

To play with parallel build I created a spike project. You can find it attached to this post or at the following svn url:

The project is a combination of the above mentioned posts at the msbuild blog with few twists:
- To avoid dependency on the "sleep" ms-dos utility I added a simple SleepTask directly to the hands-on project.
- I combined the projects structure of the first post with dependency resolution per the second link.

Project structure looks like:

 

With 1.proj being:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

  <Target Name="t">

    <Message Importance="high" Text="## starting 1 ##"/>

    <SleepTask Time="3000" />

    <Message Importance="high" Text="## finishing 1 ##"/>

  </Target>

</Project>
 

2.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

  <Target Name="t">

    <Message Importance="high" Text="## starting 2 ##"/>

    <SleepTask Time="3000" />

    <Message Importance="high" Text="## finishing 2 ##"/>

  </Target>

</Project>

3.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <UsingTask TaskName="MultiProcBuild.SleepTask" AssemblyFile="bin\Debug\MultiProcBuild.dll"/>

    <ItemGroup>

      <ProjectReference Include="2.proj"/>

    </ItemGroup>

    <Target Name="t" DependsOnTargets="BuildProjectReferences">

      <Message Importance="high" Text="## starting 3 ##"/>

      <SleepTask Time="3000" />

      <Message Importance="high" Text="## finishing 3 ##"/>

    </Target>

    <Target Name="BuildProjectReferences">

      <MSBuild Projects="@(ProjectReference)" />

    </Target>

</Project>

Than we have two flavors of "root" project, one is running 1.proj and 2.proj and we kick it off by msbuild build12.proj /m:

Whereas build123.proj builds 1,2,3 in parallel. You can see from the output of msbuild /m, that project was built as expected, taking a bit more than 6 seconds and running 2.proj in advance of 3.proj (and only once:))

Midnight approaching and this is it for today. For completeness, providing the source for remaining projects and a SleepTask:

build12.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <Target Name="t">

    <Message Importance="high" Text="## in root building children ##"/>

    <MSBuild Projects="1.proj;2.proj;" BuildInParallel="true"/>

    <Message Importance="high" Text="## in root done building ##"/>

  </Target>

</Project>

build123.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">

  <Target Name="t">

    <Message Importance="high" Text="## in root building children ##"/>

    <MSBuild Projects="1.proj;2.proj;3.proj" BuildInParallel="true"/>

    <Message Importance="high" Text="## in root done building ##"/>

  </Target>

</Project>

SleepTask:

using Microsoft.Build.Utilities;

using Microsoft.Build.Framework;

using System;

using System.Threading;

using System.Globalization;

namespace MultiProcBuild

{

    /// <summary>

    /// MSBuild task to delay the process for the required amount of time.

    /// </summary>

    /// <remarks>Realized via call to Thread.Sleep.</remarks>

    public class SleepTask : Microsoft.Build.Utilities.Task

    {

        /// <summary>

        /// Amount of time in milliseconds to sleep.

        /// </summary>

        [Microsoft.Build.Framework.Required()]

        public int Time { get; set; }

        public override bool Execute()

        {

            try

            {

                Thread.Sleep(Time);

                Log.LogMessage(MessageImportance.High, String.Format(CultureInfo.InvariantCulture,

                    "Task {0}, slept for {1} ms.", GetType().Name, Time.ToString()));

                return true;

            }

            catch (Exception ex)

            {

                Log.LogErrorFromException(ex, true);

                return false;

            }

        }

    }

}

Project url: http://code.google.com/p/toolsdotnet/source/browse/#svn/trunk/Tools.Net/spikes/Versioning/MultiProcBuild
Zipped sources: http://cid-c651f6a9f36fb87d.skydrive.live.com/self.aspx/Public/MultiProcBuild.zip