Automating the Automator – Hudson job creation - Specialmoves

Automating
the Automator - 
Hudson job creation

Specialmoves Labs

Written by Sunil Jolly,

If you don’t know about the wonders of Hudson and Continuous Integration, check out this earlier post on Hudson and Automating Development.

Why automate the automator?

For a while now we’ve been using Hudson as our CI box for our biggest client's site and it has developed to the point where we have multiple Hudson jobs for each development branch. The jobs we have for each development branch are:

  • Trigger .NET build – this triggers the .NET build when certain files/folders are updated
  • Trigger Flash build – this triggers a Flash build when certain files/folders are updated
  • Trigger redeploy after HTML site update – this triggers a deploy of static web assets when updated
  • .NET build – builds the .NET dlls
  • Flash build – compiles the SWF(s) for the site
  • CMS build – compiles the CMS Flex app
  • Site Deploy – deploys the site (this is triggered by any of the “build” jobs)

Note – we have “trigger” builds to prevent jobs running when not required (i.e. the Flash build commits a version file to the DotNet build which would trigger a DotNet job build if it was polling the entire DotNet project). The “trigger redeploy after HTML site update” job polls static assets in the DotNet project and does a quick deploy without rebuilding the DotNet project.

We often have several development branches running concurrently and setting up these jobs for each branch was becoming an overhead. Even when using Hudson’s duplicate job option, the process was slow and error prone.

So, the solution? Create a Hudson job that creates all these jobs! Yes, initially it does sound a little bit crazy … but it works.

How to build the automator

There are a few steps to get this working:

  1. Create XML config file templates for each build you want to automatically create. These are based on XML config files from existing jobs.
  2. Create an Ant task to create the new branches in SVN.
  3. Create an Ant task that creates the new (empty) Hudson jobs. We then configure those jobs using the XML files from step #1.
  4. Create the automater job. This will be a parameterised job which runs the Ant tasks we made in steps #1 and #2.
  5. Reload Hudson so it refreshes with the new jobs.

Let’s break down each step:

1. Create config file templates

Every Hudson job has a config.xml file. They can be found in each job folder in the “jobs” folder in your Hudson installation (i.e. HUDSON_HOME/jobs/[job-name]/config.xml) They contain all the information for the Job including the name, description, builders, publishers…We copied the config files for jobs we wanted to duplicate and replaced any values we wanted to select with Ant variable names surrounded by double hashes.i.e. where there was a SVN url for the dot net repos, we replaced it with ##net_svn_url##

<hudson.scm.SubversionSCM_-ModuleLocation>
  <remote>##net_svn_url##</remote>
  <local>DotNet</local>
</hudson.scm.SubversionSCM_-ModuleLocation>

2. Create Ant tasks that create new SVN branches

We used SVNAnt to create new branches in SVN. You can use variables passed from the automater job (see below).

3. Create Ant Tasks to create new jobs

Hudson has an API which does this for us. Call “createItem” with your job name. i.e.

<antcall target="_doPost">
  <param name="url" value="${server_url}/createItem?name=${dotnet_job_name}"/>
</antcall>

Note – the Ant variables will be provided by the automater job (see below)

We used this excellent script by Ciaran Jessup for the “_doPost” antcall. That post also has information on how to clone Hudson jobs.

Now we need to update the empty config file for the job by using our config file templates from step #1. This is simple with Ant:

<property name="dotnet_new_config_file_path" value="${hudson_jobs_file_path}\${dotnet_job_name}\config.xml" />
<copy file="${dotnet_config_file_path}" tofile="${dotnet_new_config_file_path}" />
<replaceregexp file="${dotnet_new_config_file_path}" match="##net_svn_url##" flags="g" replace="${dotnet_svn_branch_url}"/>

In the example above the config template is being copied to the new job folder and then any ##net_svn_url## text is replaced with the value of the $(dotnet_svn_branch_url} variable. Our Ant tasks also create version files, update app config files and deploy files. We used the same method to “inject” our variables into the config files.

4. Create the automator job

We now have Ant tasks that create jobs configured with our custom templates. Now we need the parameterised build to pass the variables our job needs to our Ant tasks. Again, this is easy with Hudson.

Make a new job and then select “This build is parameterized”. For the example above we made a String Parameter with the name “dotnet_svn_branch_url” with a default value of “trunk” (leave that blank if you want). We also created a String Parameter for “dotnet_job_name”.

The automator job needs to be configured to build the Ant task.

When the automator job runs the user is prompted to enter a a value for dotnet_svn_branch_url and it will then run the build with the variables we need. Easy!

5. Reload Hudson

Your jobs won’t appear in Hudson yet. You’ll need to use the Hudson reload function:
Note – we had to add a two second delay because the reload wasn’t happening without it. I guess its a timing issue which we’ll have to keep an eye on and investigate further.

<sleep seconds="2"/>
<antcall target="_doPost">
  <param name="url" value="${server_url}/reload"/>
</antcall>

Improvements

So that’s it all done. Now we run a parameterised build, stick in a few values and hey presto, our jobs are built and the project is ready to check out of SVN. We’ve still got more we’d like to do.

  • We want to create a tab in Hudson automatically and stick the new jobs in there
  • It’d be great to get Hudson to also create a new development environment on our dev server (that’s a seperate process at the moment)

We’ll have to leave that for another post.