Satalyst Brilliance on 28 Jul 2015

Azure Continuous Integration Monitoring Part 1

Satalyst Innovation

Azure powered multi-stage CI monitoring tools

To further encourage and promote innovation at Satalyst, we have quarterly FedEx Days where Satalyst technicians, developers, architects and the like get together to work on pet projects that are aligned with the company’s strategic direction.  The objective is to ‘deliver’ something at the end of the day – hence calling it FedEx Day.

This blog post is the result of a 2015 FedEx Day project on Continuous Integration.  This is the first in a series of 3 posts covering something fun we’ve been messing around with in our spare time. Part 1 includes a little about what we’re trying to do, part 2 describes the infrastructure we built on Azure to do all the heavy lifting, and part 3 shows the finished shiny and wraps things up.

Over the last few months, Satalyst technical folk have been building out an experimental Azure Continuous Integration monitoring system. A lot’s been written about what CI is and why you should be using it, so if it’s unfamiliar, you might start with Martin Fowler’s essay Continuous Integration. Continuous Delivery and Continuous Deployment are more recent developments, and have also had a lot of pixels and ink spent on them. Go ahead and hit Google if you’re not familiar with CD.

CI and CD tools (like TeamCity, Jenkins, and Octopus Deploy, among others) are able to send notifications to various channels when they do something, so we decided to make use of those notifications to build some monitoring tools that are independent of the choice of CI/CD tools. Our in house dev tooling is set up to use TeamCity and Octopus Deploy, so that’s what we used, but the same approach can be applied to any CI/CD build tools that can emit notifications.

We chose to model our CD pipeline as TeamCity jobs that depended on each other, so we had:

  • a build task, that triggers on VCS commit, and compiles the source, runs unit tests, packages the build artifacts, etc.;
  • an integration test job, that triggers on successful completion of the build task, and runs some extra integration tests on the previously produced artifacts;
  • a deployment job, that triggers on successful completion of the integration test job, and calls out to Octopus Deploy to deploy the artifacts into a test environment; and
  • another test job, which triggers on successful completion of the integration test job, and runs some more tests against the deployed app.

That’s as far as this particular pipeline goes, but you can imagine a longer pipeline with more jobs that might also deploy into a staging environment, runs more tests, and maybe finally deploys to production and runs more tests.

At some offices, you’ll see build lights that indicate the status of a specific CI build job, like these in the Thoughtworks offices:

"Series of build lights" by Dushan Hanuska

“Series of build lights” by Dushan Hanuska

The problem with these is that they only show a single CI job each. If you want to show the status of multiple CI jobs, then you need multiple lights, like the photo above. Worse, if those multiple jobs are part of a pipeline, they can’t easily show you the status of the pipeline as a whole. In the example above, if the Build step fails and goes red, the other 3 steps will still be green, when they really shouldn’t be.

Read on in the next installment to see how we addressed this shortcoming.