I listened to a recent episode of .NET Rocks about CI/CD pipelines and the guest had built a yeoman generator to build out the pipeline. It is specific to the Microsoft offerings of Visual Studio Team Services and Team Foundation Services, but it is an interesting idea. There isn’t a reason you couldn’t build something similar on top of Travis, Circle, or Semaphore. I had touched on some of the other generators at work previously, but those were bootstrapping service and application code, not infrastructure. But that’s the advantage of infrastructure as code – you get all of the toolchain available to work with it.
At work we have code to build out the CI/CD pipelines but it is more of a desired state thing than a generator. It’s all built on top of a local Jenkins install, and regularly recreates all of the job definitions to prevent any changes made by hand from living very long. At previous jobs the automation for building different services and libraries were all a little different or built in a brittle template/multipart style that didn’t allow for any changes in a safe way. Our tool has been great for keeping all of the microservice jobs the same, templated in a way that makes upgrading them in groups easy enough, and even fairly testable.
The testing aspect is part of what’s most interesting, we boot up a Jenkins instance inside a container and generate the jobs into it, with some different configuration for notifications and where to put artifacts generated to blackhole all of it. The deployment jobs work using the same version running to the dev cluster so it should all be idempotent. The biggest issue I’ve seen so far is that it runs a ‘representative sample’ of jobs as part of testing; that hasn’t always been as representative as we would want it to be especially as we add new jobs. We’ve got roughly 300 repos but only maybe 150-200 use managed builds like this. The good news is they break much less often than the unmanaged ones, but the bad news is when they break it’s usually a large swath of them that go down all at once.
AWS CodeStar was recently announced as a hosted version of this sort of tool. It creates repos, scaffolds a new application, can deal with permissions, and carries through to the provisioning of hosting as well. It has options for multiple languages and hosting options. So it seems like it is simplifying a lot of the initial setup of a service.
I’m not sure where the tipping point on the value of setting up your own infrastructure would be, but it seems like at least 100 repos depending on the complexity of each build. This sort of automation will become more important as microservices deployment continues to get bigger. If you haven’t done any sort of automation like this it’s worth looking into depending on the scope of the project.