MaestroDev named DevOps “Cool Vendor” by Gartner

MaestroDev logo

Warning, some self-promotion ahead! :)

Gartner has published their annual list of Cool Vendors, including a section for DevOps, where we are one of the 5 selected companies.

Not a big fan of this analyst things, but quite proud of being included in such a short list, right next to the people from CFEngine, Opscode and Puppet Labs, that are very active on the DevOps space and, in the case of PuppetLabs, whose products we use heavily for automation.

MaestroDev, an innovation leader in DevOps Orchestration, has been included in the list of “Cool Vendors in DevOps, 2012” report by Gartner, Inc.

And thanks to our great customers too!

Keith Campbell, CTO, Informatics, said “The Maestro product has automated our build process all the way through packaging. We are using our same toolset, but the Maestro Composition engine gives us consistency and speed that we did not have before. With Maestro, we are planning our development-cloud environment as well — reducing our build cost even further because we can dynamically integrate hybrid resources and external services into our workflows.”

You can check out the rest of the press release at the MaestroDev blog, and the Gartner Cool Vendors report.

Automatically download and install VirtualBox guest additions in Vagrant

So, are you already using Vagrant to manage your VirtualBox VMs?

Then you probably have realized already how annoying is to keep the VBox guest additions up to date in your VMs.

Don’t worry, you can update them with just one command or automatically on each start using the Vagrant-vbguest plugin.

Installation

Requires vagrant 0.9.4 or later (including 1.0)

Since vagrant v1.0.0 the prefered installation method for vagrant is using the provided packages or installers.

Therefore if you installed Vagrant as a package (rpm, deb, dmg,…)

vagrant gem install vagrant-vbguest

Or if you installed vagrant using RubyGems (gem install vagrant):

gem install vagrant-vbguest

Usage

By default the plugin will check what version of the guest additions is installed in the VM every time it is started with vagrant start. Note that it won’t be checked when resuming a box.

In any case, it can be disabled in the Vagrantfile

Vagrant::Config.run do |config|
  # set auto_update to false, if do NOT want to check the correct additions
  # version when booting this machine
  config.vbguest.auto_update = false
end

If it detects an outdated version, it will automatically install the matching version from the VirtualBox installation, located at

  • linux : /usr/share/virtualbox/VBoxGuestAdditions.iso
  • Mac : /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
  • Windows : %PROGRAMFILES%/Oracle/VirtualBox/VBoxGuestAdditions.iso

The location can be overridden with the iso_path parameter in your Vagrantfile, and can point to a http server

Vagrant::Config.run do |config|
  config.vbguest.iso_path = "#{ENV['HOME']}/Downloads/VBoxGuestAdditions.iso"
  # or
  config.vbguest.iso_path = "http://company.server/VirtualBox/$VBOX_VERSION/VBoxGuestAdditions.iso"
end

If you have disabled the automatic update, it still easy to manually update the VirtualBox Guest Additions version, just running from the command line

vagrant vbguest

Learning Puppet or Chef? Check out Vagrant!

If you are starting to use Puppet or Chef, you must have Vagrant.

Learning Puppet can be a tedious task, getting up the master, agents, writing your first manifests,… A good way to start is using Vagrant, an Oracle VirtualBox command line automation tool, that allows easy Puppet and Chef provisioning on VirtualBox VMs.

Vagrant projects are composed by base boxes, specifically configured for Vagrant with Puppet/Chef, vagrant username and password, and anything else you may want to add, plus the configuration to apply to those base boxes, defined with Puppet or Chef. That way we can have several projects using the same base boxes shared where the only difference are the Puppet/Chef definitions. For instance a database VM and a web server VM can both use the same base box and just have different Puppet manifests, and when Vagrant starts them, it will apply the specific configuration. That also allows to share boxes and configuration files across teams, for instance having one base box with the Linux flavor used in a team, we can just have in source control the Puppet manifests to apply for the different configurations that anybody from Operations to Developers can use.

There is a list of available VMs or base boxes ready to use with Vagrant at www.vagrantbox.es. But you can build your own and share it anywhere, as they are just (big) VirtualBox VM files, easily using VeeWee, or changing a base box and rebundling it with vagrant package.

Usage

Once you have installed Vagrant and VirtualBox.

Vagrant init will create a sample Vagrantfile, the project definition file that can be customized.

$ vagrant init myproject

Then in the Vagrantfile you can change the default box settings, and add basic Puppet provisioning

config.vm.box = "centos-6"
config.vm.box_url = "https://vagrant-centos-6.s3.amazonaws.com/centos-6.box"

config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "manifests"
  puppet.manifest_file = "site.pp"
end

In manifests/site.pp you can try any puppet manifest.

file { '/etc/motd':
  content => 'Welcome to your Vagrant-built virtual machine! Managed by Puppet.\n'
}

Vagrant up will download the box the first time, start the VM and apply any configuration defined in Puppet

$ vagrant up

vagrant ssh will open a shell into the box. Under the hood vagrant is redirecting the host port 2222 to the vagrant box 22

$ vagrant ssh

The vm can be suspended and resumed at any time

$ vagrant suspend
$ vagrant resume

and later on destroyed, which will delete all the VM files.

$ vagrant destroy

And then we can start again from scratch with vagrant up getting a completely new vm where we can make any mistakes :)

Introduction to Puppet

Enough about philosophical posts, let’s get started with some practical Puppet.

Manifests

Puppet configuration files are called manifests, written in a ruby-like DSL. Puppet provides types and functions to manage typical resources (files, services, users, groups,…) and new ones can be defined through extensions called modules.

The standard types that can be used are listed in the Puppet reference. There is a cheat sheet available (pdf) with the main ones.

The resources are grouped in classes, that can later be easily reused.

class 'maven' {
  exec { 'maven-untar':
    command => 'tar xf /tmp/x.tgz',
    cwd     => '/opt',
    creates => "/opt/apache-maven-${version}",
    path    => ["/bin"],
  } ->
  file { '/usr/bin/mvn':
    ensure => link,
    target => "/opt/apache-maven-${version}/bin/mvn",
  }
  file { '/usr/local/bin/mvn':
    ensure  => absent,
    require => Exec["maven-untar"],
  }
  file { "${home}/.mavenrc":
    mode    => '0600',
    owner   => $user,
    content => template('maven/mavenrc.erb'),
    require => User[$user],
  }
}

Infrastructure IS code, for example we can specify that we want the openssh-server package installed

package { 'openssh-server':
  ensure => present,
}

Declarative model

Puppet uses a declarative model, where we define state, not process. We define that a service must be running and puppet will start it if not running, or do nothing if it already is.

service { 'ntp':
  name   => 'ntpd',
  ensure => running,
}

There is no scripting, we don’t make the service start, just define whether it should be running. This is key to understand how puppet works. A side effect is that variables can only be assigned once, so they are pretty much like constants.

Architecture

Puppet is arranged in a master – agent architecture.  The master serves the manifests and files, and the agents poll the master at specific intervals of time to get their configuration. The master does not push anything into the client.

Agents identify with the master using SSL, so the first time an agent tries to connect to the master, the agent certificate needs to be approved (in the default configuration), and that’s usually a source of problems.

File structure

Puppet configuration files are usually in /etc/puppet.

The main files in there are manifests/site.pp which defines the configurations, and the manifests/nodes.pp that defines how those configurations apply to the different nodes or agents, based on their hostname, generally, or other properties.

Site

class 'dave' {
  user { 'dave':
    ensure     => present,
    uid        => '507',
    gid        => 'admin',
    shell      => '/bin/zsh',
    home       => '/home/dave',
    managehome => true,
  }
  file {'/tmp/test1':
    ensure  => present,
    content => "Hi.",
  }
}

Nodes

node 'someserver.domain.com' {
  class { 'dave': }
}

More information

More information about types, resources, manifests, variables,… at learning puppet from PuppetLabs.

Infrastructure as Code

DevOps is not about the tools

That’s true, in the same way that agile is not about the tools either, it’s a set of ideas, concepts, best practices,…

Nice, but… how can I successfully implement it?

Tools can enable change in behavior and eventually change culture [Patrick Debois]

Printer in 1568

The same way the Guttemberg printer was a tool that enabled a cultural change, or that Agile development wouldn’t be possible without Continuous Integration servers, DevOps relies on some tools to implement its principles.

Unfortunately, the same way everyone thinks of themselves as being intelligent enough, and every tool out there is magically cloud enabled, now every tool claims to be DevOps.

However, there is agreement that tools that allow us to deal with infrastructure as code are key on implementing DevOps concepts.

It’s all been invented already, now it’s standardized with tools like Chef or Puppet. Before, you could write your own scripts to automate server installation, configuration,… but everyone would do it their very own way.

Now there’s some common language used by Puppet or by Chef, that allows to share and reuse configuration as modules or recipes.

Infrastructure as Code, a key concept

The concept that infrastructure should be treated as code is really powerful. Server configuration, packages installed, relationships with other servers,… should be modeled with code to be automated and have a predictable outcome, removing manual steps prone to errors. Doesn’t sound bad, does it?

But new solutions bring new challenges, and when infrastructure is code we face the same problems faced by developers.

  • What version of the infrastructure are we using in production?
  • how can we ensure that when an issue is found it gets fixed and redeployed?
  • how can we test the infrastructure as we develop it?

That’s why when dealing with infrastructure as code we should follow development best practices.

For instance we can (and should!)

  • tag, branch and release the code that define our servers.
  • have a lifecycle that covers different stages through the infrastructure code, ie. dev, QA, production.
  • continuously test our infrastructure as we make changes.

Is DevOps killing the Operations team?

To make error is human. To propagate error to all server in automatic way is DevOpsHearing everywhere about DevOps and how it is all about automation, and how manual steps should be removed from Operations. Starting to worry about your OPs job?

On one hand, yes, you should worry.

My job is to make other people’s jobs unnecessary.

While I was working on Maven the goal was to automate and standardize all the build steps so there’s no more need to have a magician build master that is the only one that knows how to build the software. All Maven projects are built in the same way and there’s no need to do any manual step. That ended the build master job in many companies as they knew it. Those that were interested enough moved on to do more useful tasks, like setting up continuous integration servers, integrating new quality assurance tools, adding metrics,…

So, on the other hand, no, you shouldn’t worry as long as you want to explore new areas, because there’s still plenty to improve. Stop doing tedious manual tasks and focus on what’s really important.

You should just worry about the NOOPS guys ;)

DevOps: how we got here

Developer toolset

From the developer point of view, there are some tools involved in the source-to-deploy process

  • Source control management tools: Subversion, Git, Mercurial, Perforce,…
  • Build tools: Maven, Ant, Ivy, Buildr, Graddle, Rake,…
  • Continuous Integration tools: Continuum, Jenkins, Hudson, Bamboo,…
  • Repository (Artifact) management tools: Archiva, Nexus, Artifactory,…

The #1 programmer excuse for legitimately slacking off: My code is compiling

When everything is set together, we can have a CI schedule that is building automatically the changes from the SCM as they are committed, deploying to an artifact repository the result of the build or sending a notification if there is any error. Everything fully automated. A change is made to SCM, the CI server kicks in, builds and runs all sort of tests (unit, functional, integration,…) while you go off for a sword fight with your coworkers.

Now what? somebody sends by email the tarball, zipfile,… to the operations team? oh, no that would be too bad. Just send them the url to download it… And even better send some instructions, a changelog, upgrade task list,…

What developers do today to specify deployments and target environments is not enough. 

The simplest solutions are often the cleverest. They are also usually wrong.

Using tools like Maven in the Java world or Bundle in Rubyland you can explicitly list all the dependencies and versions you need. But there are some critical dependencies that are never set.

It is just too simple.

Packages installed, C libraries, databases, all sort of OS and service level configuration,… That’s the next level of dependencies that should be explicitly listed and automated.

For example, think about versions of libc, postgresql, number of connections allowed, ports opened, opened file descriptors limit,…

Operations

Requirements

From the point of view of the operations team the number of requirements is complex: operating system, kernel version, config files, packages installed,…

And then multiply that for several stage configurations that most likely won’t have the exact same configurations.

  • dev
  • QA
  • pre-production
  • production

Deployment

Deployment of the artifacts produced by the development team is always a challenge

  • How do I deploy this?
  • Reading the documentation provided by the development team?
  • Executing some manual steps?

That is obviously prone to errors

Cloud

It’s nothing new but it has increased with the proliferation of Cloud based environments, making it easier and easier to run dozens or hundreds of servers at any point in time. Even knowing how to deploy to one server, how is it deployed to all those servers? what connections need to be established between servers? how is it going to affect the network?

FOSDEM

The slides from my From Dev to DevOps talk at FOSDEM 2012 Brussels are up in Slideshare. The material is also in the Lanyrd page.

The conference was huge, I’ve heard that over 4000 people showed up this year, for this free (as in both software and beer) event, organized on several tracks. Not bad considering that the temperature was between -15 and -6C (5 and 23 Fahrenheit) and got a lot of snow.

I spoke on the main auditorium with capacity for 1400 people, there was plenty of space. Unfortunately (for me, but great for the other speakers and the organization) some other devrooms got filled quickly, and couldn’t get into the Configuration and Systems Management Devroom, which should have been really good as nobody left as I was waiting outside between talks :(

A lot of care by the organization, including a note with a 24/7 phone number in case I got “lost/confused/arrested” (which makes me think it must have happened before), and a full commitment to promote belgian beer ;)

And thanks to the people that attended my talk and engaged through twitter, got great feedback and hints for improvements on the future of development and infrastructure automation!

Grand Place 360 panorama

“From Dev to DevOps” at the Paris JUG next week, FOSDEM Brussels in February

The year starts with two speaking engagements, about development and DevOps

Soirée Devops @ Paris JUG

Next week, Tuesday January 10th at the I.S.E.P. Paris

An afternoon about DevOps, with Henri Gomez also speaking, where I’ll be finally able to put faces to some long time online friends and collaborators on different open source projects.

FOSDEM

Sunday February 5th at the ULB Campus Solbosch in Brussels.

the biggest free and non-commercial event organized by and for the community

Interesting conference concept with no registration necessary, around 4000 attendees in 4-6 tracks, completely free, including a Friday Belgium beer event, as they say don’t miss this great opportunity to taste some of the finest beer in Belgium (and, hence, in the world). And they even include spouse/partner tours!

Are you going to any of them? say hi!

About: DevOps

This is the start of a (hopefully) series of posts about DevOps, based on my presentation From Dev to DevOps.

The Agile movement stablished a series of development practices quite common nowadays, or at least highly desired:

  • planning
  • iterative development
  • continuous integration
  • release soon, release often

But what happens after the development cycle? The process of building software or products does not stop there. Even if you are an agile shop, you may hit a wall when trying to move between Development, QA and Operations.

DevOps is a door in that wall.

DevOps is the intersection between Development, QA and Operations. You can also think of it as DevQAOps.

Old issues fixed, new issues come up

As some issues were solved by Agile methodologies, new issues arose. Agile development generates a higher number of releases (release early/release often) than waterfall methodologies, and that pushes the problem from development teams to operations, that have to deal with more frequent deliveries to production.

DevOps addresses

Fear of change

Once a product is working, changes are seen as a risk by business, developers, QA, operations,… Reducing the time to make (and revert!) changes increases the assurance to do more frequent changes.

Risky deployments

Automation, automation, automation! Not only the development needs to be automated. No more docs with instructions, manual steps, individual knowledge, “deployment master” or hacks. All the steps up to production need to be automated too, including deployments, to reduce risk of manual mistakes.

It works on my machine!

With the proliferation of virtualization, cloud, PaaS, stacks,… is absolutely necessary to be able to replicate environments for development, QA, pre-production,… Ensuring that proper level of testing is conducted as the product (software + environment) progresses through the lifecycle.

Siloisation

Teams split in Dev, QA, Ops,… cause walls to stand between them, and unnecessary friction, with each team pushing for its own benefit and requiring escalation of issues to account for the overall business good. Getting all the teams on the same table more often the friction is reduced.

Dev Change vs. Ops stability

Developers are usually measured by the number of features implemented, while operation teams are measured by the performance and stability on production. Two different goals that make both teams clash in many occasions. Bring issues early on and involve all the teams in the decisions.

Read more at What Is This Devops Thing, Anyway?

DevOps is a extension of Agile

Does this sound familiar?

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Exactly, that’s the Agile manifesto, which is still perfectly valid, with some clarifications, if we get into detail:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software products.
  • Welcome changing requirements, even late in development any part of the DEV/QA/OPs cycle.  Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software products frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Business people and developers development/QA/Operations teams must work together daily throughout the project.
  • The most efficient and effective method of conveying information to and within a development DEV/QA/OPs team is face-to-face conversation.
  • Agile processes promote sustainable development. The sponsors, developers, QA and OPs teams, and users should be able to maintain a constant pace indefinitely.

What does that exactly mean

  • software/working software. The concept of product, meaning software running, in production
  • late in development. Late in the development, QA or operations phases, not just development
  • developers. Don’t consider only developers, QA and operations teams need to be involved too

The key part is the involvement of not just the development team but the QA and Operations teams.

Game theory

Game theory is defined as the method for analyzing calculated circumstances, such as in games, where a person’s success is based upon the choices of others. If you have watched A beautiful mind, you’ll remember the scene where he tries to pick up a girl at a bar.

What does it have to do with DevOps? The success of the Dev team greatly depends on actions done by the Ops team, good software deployed incorrectly will transmit the wrong impression. Likewise, the success of the Ops team definitely depends on the work of the Dev team, as poorly developed software will cause all sort of performance issues.

If all teams try to achieve their own benefit, it’s not necessarily a good thing for the overall business. Imagine the Ops team not pushing new features because they don’t want to risk the production stability. Or the Dev team pushing new features to production without proper testing because it’s a QA problem, not theirs, and are only measured on the amount of features pushed.