Notes from: Continuous Integration for Infrastructure

Continuous integration is a mechanism to test every commit in a central place, collectively as a team. It encourages good team work since it is the central place for every team member’s work to integrate.

Infrastructure is now seen as code, and someone else deals with the hardware and bare metal. So we should apply tried and tested software engineering practices to the infrastructure domain.

The 5 off-the-shelf things we should bring over:

  • Configuration management
  • Acceptance testing
  • Verifying machine images
  • Load testing
  • Provisioning

Both chef and puppet have good unit and integration testing support. We want to provision a short-lived virtual machine and make assertions against the resulting state.

Verifying Machine Images

Your virtual images need a pipeline as well. You can use Packer to create images for multiple providers from a single JSON configuration file. Packer will create the images for me, but how do I know if that image works?

Like before, we can fall back to using serverspec, and only if the tests pass do we save the image.

Load Testing

Load testing is often ad hoc. And because most organizations are distributed, it becomes a more difficult problem to simulate. But ultimately, nothing is really good for automated load testing to put into the build pipeline with good output.

Gatling is preferred, which is a Scala load testing tool. Aggregate assertions are the killer feature of Gatling. What makes this better than other load testing tools is that the run can fail and hook into your overall suite besides just throwing a bunch of load at your site.

Continuous load testing supplements rather than replaces ad hoc load testing. Situations like long-running load testing is something that is impractical for an every-commit run. Though it does make ad hoc load testing much easier to do.

Provisioning

Ansible is a great tool for describing and orchestrating virtual machines in different environments. Use bats for testing bash shell scripts. Bats is usually a good thing to use in combination with serverspec. So we can use ansible to manage multiple virtual machines, bats to test that we do in fact have the correct number provisioned, and serverspec to test that they have been provisioned correctly. Serverspec has a mode where it can run against a remote host.

Conclusion

This is where infrastructure needs to head: to describe everything in a declarative way – either in a yaml file or json file – then we can put this into a single pipeline and collaborate around it with continuous integration. Pipelines is the core idea here.

This is also the beginning, other ideas in software configurations, like feature flags, or testing software-defined networks. We need more devs to get interested in networking. Also, moving infrastructure code from continuous integration to continuous delivery.