One of the main ideas behind DevOps adoption is the concept of  “infrastructure as code”. Tools like Puppet or Chef allow you to programmatically define your infrastructure, the provisioning of your servers: what packages are installed, what is the content of files,…

If server provisioning is a key point in operations, then code management becomes key too once you start coding your servers. You need source control for your infrastructure, you need tags, versioning, dependencies between components,… You need development, testing, QA, release,… for your infrastructure!

Imagine a environment where you have some server stack running in production using Puppet, with a manifest that defines packages and files in that server, and many servers running the same configuration. That Puppet definition must be in source control.

Now a security fix or new version of package must be installed in all the servers, do you just want to change the manifest and push it out to all the running servers? doesn’t sound like a great idea, does it? Hey, we have been tuning development best practices over the years for use cases just like this one.

What you want to do is create a new branch where you can do that change, and test it in some server that is not in production, let’s call it development environment, original isn’t it?
The change works as expected, your app still works, great! now you can probably merge that branch of the Puppet manifests into trunk, with possibly other changes made by other people, that at some point you will want to test together, in a production-like environment, maybe with several servers in a cluster, load balancing, etc… and very importantly, with the next version of the application that is going to be deployed. You create a new tag and version to be able to identify it later and deploy to that environment, let’s call it QA or staging.

What all this cycle allows you to do is clearly define what is running in each environment, using versions, and easily find issues between deployments, using source control, being able to roll back to known working configuration if needed.

After all, if you deal with infrastructure as code you should use code development best practices, and you’ll get the same benefits.