Monthly Archives: March 2012

Learning Puppet or Chef? Check out Vagrant!

Posted on by Carlos Sanchez
1

If you are starting to use Puppet or Chef, you must have Vagrant.

Learning Puppet can be a tedious task, getting up the master, agents, writing your first manifests,… A good way to start is using Vagrant, an Oracle VirtualBox command line automation tool, that allows easy Puppet and Chef provisioning on VirtualBox VMs.

Vagrant projects are composed by base boxes, specifically configured for Vagrant with Puppet/Chef, vagrant username and password, and anything else you may want to add, plus the configuration to apply to those base boxes, defined with Puppet or Chef. That way we can have several projects using the same base boxes shared where the only difference are the Puppet/Chef definitions. For instance a database VM and a web server VM can both use the same base box and just have different Puppet manifests, and when Vagrant starts them, it will apply the specific configuration. That also allows to share boxes and configuration files across teams, for instance having one base box with the Linux flavor used in a team, we can just have in source control the Puppet manifests to apply for the different configurations that anybody from Operations to Developers can use.

There is a list of available VMs or base boxes ready to use with Vagrant at www.vagrantbox.es. But you can build your own and share it anywhere, as they are just (big) VirtualBox VM files, easily using VeeWee, or changing a base box and rebundling it with vagrant package.

Usage

Once you have installed Vagrant and VirtualBox.

Vagrant init will create a sample Vagrantfile, the project definition file that can be customized.

$ vagrant init myproject

Then in the Vagrantfile you can change the default box settings, and add basic Puppet provisioning

config.vm.box = "centos-6"
config.vm.box_url = "https://vagrant-centos-6.s3.amazonaws.com/centos-6.box"

config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "manifests"
  puppet.manifest_file = "site.pp"
end

In manifests/site.pp you can try any puppet manifest.

file { '/etc/motd':
  content => 'Welcome to your Vagrant-built virtual machine! Managed by Puppet.\n'
}

Vagrant up will download the box the first time, start the VM and apply any configuration defined in Puppet

$ vagrant up

vagrant ssh will open a shell into the box. Under the hood vagrant is redirecting the host port 2222 to the vagrant box 22

$ vagrant ssh

The vm can be suspended and resumed at any time

$ vagrant suspend
$ vagrant resume

and later on destroyed, which will delete all the VM files.

$ vagrant destroy

And then we can start again from scratch with vagrant up getting a completely new vm where we can make any mistakes 🙂

Introduction to Puppet

Posted on by Carlos Sanchez
Reply

Enough about philosophical posts, let’s get started with some practical Puppet.

Manifests

Puppet configuration files are called manifests, written in a ruby-like DSL. Puppet provides types and functions to manage typical resources (files, services, users, groups,…) and new ones can be defined through extensions called modules.

The standard types that can be used are listed in the Puppet reference. There is a cheat sheet available (pdf) with the main ones.

The resources are grouped in classes, that can later be easily reused.

class 'maven' {
  exec { 'maven-untar':
    command => 'tar xf /tmp/x.tgz',
    cwd     => '/opt',
    creates => "/opt/apache-maven-${version}",
    path    => ["/bin"],
  } ->
  file { '/usr/bin/mvn':
    ensure => link,
    target => "/opt/apache-maven-${version}/bin/mvn",
  }
  file { '/usr/local/bin/mvn':
    ensure  => absent,
    require => Exec["maven-untar"],
  }
  file { "${home}/.mavenrc":
    mode    => '0600',
    owner   => $user,
    content => template('maven/mavenrc.erb'),
    require => User[$user],
  }
}

Infrastructure IS code, for example we can specify that we want the openssh-server package installed

package { 'openssh-server':
  ensure => present,
}

Declarative model

Puppet uses a declarative model, where we define state, not process. We define that a service must be running and puppet will start it if not running, or do nothing if it already is.

service { 'ntp':
  name   => 'ntpd',
  ensure => running,
}

There is no scripting, we don’t make the service start, just define whether it should be running. This is key to understand how puppet works. A side effect is that variables can only be assigned once, so they are pretty much like constants.

Architecture

Puppet is arranged in a master – agent architecture.  The master serves the manifests and files, and the agents poll the master at specific intervals of time to get their configuration. The master does not push anything into the client.

Agents identify with the master using SSL, so the first time an agent tries to connect to the master, the agent certificate needs to be approved (in the default configuration), and that’s usually a source of problems.

File structure

Puppet configuration files are usually in /etc/puppet.

The main files in there are manifests/site.pp which defines the configurations, and the manifests/nodes.pp that defines how those configurations apply to the different nodes or agents, based on their hostname, generally, or other properties.

Site

class 'dave' {
  user { 'dave':
    ensure     => present,
    uid        => '507',
    gid        => 'admin',
    shell      => '/bin/zsh',
    home       => '/home/dave',
    managehome => true,
  }
  file {'/tmp/test1':
    ensure  => present,
    content => "Hi.",
  }
}

Nodes

node 'someserver.domain.com' {
  class { 'dave': }
}

More information

More information about types, resources, manifests, variables,… at learning puppet from PuppetLabs.

Infrastructure as Code

Posted on by Carlos Sanchez
10

DevOps is not about the tools

That’s true, in the same way that agile is not about the tools either, it’s a set of ideas, concepts, best practices,…

Nice, but… how can I successfully implement it?

Tools can enable change in behavior and eventually change culture [Patrick Debois]

Printer in 1568

The same way the Guttemberg printer was a tool that enabled a cultural change, or that Agile development wouldn’t be possible without Continuous Integration servers, DevOps relies on some tools to implement its principles.

Unfortunately, the same way everyone thinks of themselves as being intelligent enough, and every tool out there is magically cloud enabled, now every tool claims to be DevOps.

However, there is agreement that tools that allow us to deal with infrastructure as code are key on implementing DevOps concepts.

It’s all been invented already, now it’s standardized with tools like Chef or Puppet. Before, you could write your own scripts to automate server installation, configuration,… but everyone would do it their very own way.

Now there’s some common language used by Puppet or by Chef, that allows to share and reuse configuration as modules or recipes.

Infrastructure as Code, a key concept

The concept that infrastructure should be treated as code is really powerful. Server configuration, packages installed, relationships with other servers,… should be modeled with code to be automated and have a predictable outcome, removing manual steps prone to errors. Doesn’t sound bad, does it?

But new solutions bring new challenges, and when infrastructure is code we face the same problems faced by developers.

  • What version of the infrastructure are we using in production?
  • how can we ensure that when an issue is found it gets fixed and redeployed?
  • how can we test the infrastructure as we develop it?

That’s why when dealing with infrastructure as code we should follow development best practices.

For instance we can (and should!)

  • tag, branch and release the code that define our servers.
  • have a lifecycle that covers different stages through the infrastructure code, ie. dev, QA, production.
  • continuously test our infrastructure as we make changes.

Is DevOps killing the Operations team?

Posted on by Carlos Sanchez
2

To make error is human. To propagate error to all server in automatic way is DevOpsHearing everywhere about DevOps and how it is all about automation, and how manual steps should be removed from Operations. Starting to worry about your OPs job?

On one hand, yes, you should worry.

My job is to make other people’s jobs unnecessary.

While I was working on Maven the goal was to automate and standardize all the build steps so there’s no more need to have a magician build master that is the only one that knows how to build the software. All Maven projects are built in the same way and there’s no need to do any manual step. That ended the build master job in many companies as they knew it. Those that were interested enough moved on to do more useful tasks, like setting up continuous integration servers, integrating new quality assurance tools, adding metrics,…

So, on the other hand, no, you shouldn’t worry as long as you want to explore new areas, because there’s still plenty to improve. Stop doing tedious manual tasks and focus on what’s really important.

You should just worry about the NOOPS guys 😉