Monthly Archives: January 2013

Testing puppet modules

Posted on by Carlos Sanchez
2

Puppet Labs logoThere are several steps depending on how much involved the tests are, what parts are tested and, of course, how long it takes to run the tests.

For unit testing we use rspec puppet, and we can check that our manifests and modules compile and contain the expected values. It can be used to test that specific types, classes or definitions are in the compiled catalog and that the parameters math the expectations.

Later on we can do some integration testing starting a new VM with Vagrant and checking that there are no errors in the provisioning, as well as checking that some conditions are met.

For rspec-puppet, PuppetLabs has created a project called puppetlabs_spec_helper that let’s us avoid writing a bunch of boilerplate. A missing point though is that it only allows to use modules for testing from git. If you’re already using librarian-puppet (and you should!) you can easily use the same Puppetfile for deploying modules and to test them. Doing otherwise sounds like a bit of useless testing, you could end with different versions in different development machines, CI server, puppet master,… So just add a call to librarian puppet in your rakefile to populate the rspec-puppet fixtures before running the specs.

Unfortunately rspec-puppet doesn’t work with Puppet 3.0.x and  at least Puppet 3.1.0-rc1 is required. It was a bit of a setback when we moved to Puppet 3 and started using hiera, which is proving to be very useful to have simpler manifests and external data injected for our Maestro installations with Puppet from scratch.

You can also use the same Puppetfile to start Vagrant boxes with the exact same version of the modules. We are using Cucumber and Aruba to execute vagrant, provision the VM with puppet and check several things, like open ports, services up,… but that’s a different story 🙂

Example

In this puppet-for-java-devs project you will find the bits that showcase all these tools integrated. It includes definition of a 3-tier system with Puppet definitions for a postgresql database, tomcat nodes with a war installed and apache nodes fronting them.

Install all required gems

bundle install

Install all Puppet modules with Puppet Librarian

librarian-puppet install

Run the specs with puppet-rspec

bundle exec rake

Start all the vms with Vagrant

vagrant up

Rakefile

require 'bundler'
Bundler.require(:rake)
require 'rake/clean'

require 'puppetlabs_spec_helper/rake_tasks'

CLEAN.include('modules', 'spec/fixtures/', 'doc')
CLOBBER.include('.tmp', '.librarian')

task :librarian_spec_prep do
 sh "librarian-puppet install"
end
task :spec_prep => :librarian_spec_prep

task :default => [:spec]

Puppetfile for librarian-puppet

forge 'http://forge.puppetlabs.com'

mod 'puppetlabs/java', '0.1.6'
mod 'puppetlabs/apache', '0.4.0'
mod 'inkling/postgresql', '0.2.0'
mod 'puppetlabs/firewall', '0.0.4'
mod 'tomcat', :git => 'https://github.com/carlossg/puppet-tomcat.git', :ref => 'centos'
mod 'maestrodev/maven', '1.x'
mod 'stahnma/epel', '0.0.2'
mod 'maestrodev/avahi', '1.x'
mod 'other', :path => 'mymodules/other'

tomcat_spec.rb with rspec-puppet

require 'spec_helper'

describe 'tomcat1.acme.com' do
  let(:facts) { {:osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => 6.3} }

  it { should contain_class('java').with_distribution /openjdk/ }

  it "configure webapp" do
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT.war')
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT/WEB-INF/lib/postgresql-9.1-901.jdbc4.jar')
  end
end

Vagrantfile

Vagrant::Config.run do |config|
  config.vm.box = "CentOS-6.3-x86_64-minimal"
  config.vm.box_url = "https://dl.dropbox.com/u/7225008/Vagrant/CentOS-6.3-x86_64-minimal.box"

  config.vm.customize ["modifyvm", :id, "--rtcuseutc", "on"] # use UTC clock https://github.com/mitchellh/vagrant/issues/912

  # db server
  config.vm.define :db do |config|
    config.vm.host_name = "db.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "db"] # name for VirtualBox GUI
    config.vm.forward_port 5432, 5432
    config.vm.network :hostonly, "192.168.33.10"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # tomcat server
  config.vm.define :tomcat1 do |config|
    config.vm.host_name = "tomcat1.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "tomcat1"] # name for VirtualBox GUI
    config.vm.forward_port 8080, 8081
    config.vm.network :hostonly, "192.168.33.11"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # web server
  config.vm.define :www do |config|
    config.vm.host_name = "www.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "www"] # name for VirtualBox GUI
    config.vm.forward_port 80, 8080
    config.vm.network :hostonly, "192.168.33.12"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

end

Managing Puppet modules with librarian-puppet

Posted on by Carlos Sanchez
14

Puppet Labs logoOnce you have your Infrastructure defined as code you need to use coding best practises.

This definitely applies to Puppet code, you have Puppet classes to encapsulate functionality, and modules that group classes together for reuse and redistribution. Modules can be reused from git repositories or from the Puppet Forge, a web archive of tarballed modules in the same fashion as rpm or maven repositories. Modules in the forge are defined by a provider, module name and version, much like Maven groupid, artifactid and version, and a provider has full control over the modules that can be published under its name, with a more open approach, as anybody can upload anything themselves through the forge website, just signup and start uploading them.

But trouble starts when you are using a number of modules from different places, as each module can have its own set of dependencies. How can the dependencies be defined, downloaded and installed automatically as happens with any other package manager?

There are several options actually

  • Git submodules. This is the poor man’s version, where you add the git repositories of other modules as git submodules and use git submodule init and update commands to stay up to date
  • Puppet module tool. Allows the installation of modules from the Puppet forge based on the module name provider/name and version. It will download and unpackage the module in the directory specified
  • Librarian-puppet. It is an extension of ruby’s bundler model to install gems, having a file that defines your dependencies, Gemfile in the case of bundler, Puppetfile in the case of librarian, and a .lock file with the resolved dependencies.

The resolution model is different than those of yum, apt-get or maven, that resolve dependencies at every run on the client, relying on the clients to consistently do that over versions and machines. In the bundler or librarian model, resolution only happens once and it’s saved to the lock file, which is used from there on by the clients until dependencies are changed.

Dependency declaration

The librarian Puppetfile allows defining module dependencies from several sources

Puppet Forge

Using the forge provider and module name, and optionally version it calls puppet module tool to fetch the tarball and extract it in the designed directory

mod "maestrodev/maven", "1.0.0"

Git

Modules can also be cloned from a git repository optionally defining what branch, tag or revision to checkout

mod "maven",
  :git => "https://github.com/maestrodev/puppet-maven.git",
  :ref => 'v1.0.0'

Path folders

Useful mostly for testing or while transitioning to a full dependency model, local paths can be used as modules

mod 'mymodule', :path => './private/mymodule'

Improvements and issues

Librarian offers several improvements over the puppet module tool. Besides being able to bring in modules from different sources, librarian locks the versions of modules used, which is a must in order to consistently reproduce results. Let’s say you define a dependency on module maestrodev/maven > 1.0.0. If you run it now then librarian will lock the version to 1.0.0. If 1.1.0 is released later your dependencies will stay the same. If you try to run the puppet module tool with

puppet module install maestrodev-maven

you would end with 1.0.0 now and  a newer version later on.

There are some problems, and we hit (and fixed!) a lot of them. We have a fork of the original project where we keep updating the maestrodev branch while our pull requests don’t make it to the original repo. Unfortunately some of the patches have been sitting there for too long and we went ahead and created the librarian-puppet-maestrodev gem  that you can use instead of the original one.

Some of the issues you’ll likely find are the mismatch between puppet module version conventions and gem versions. librarian uses the bundler gemfile resolution mechanism which does not accept some versions such as the ones including dashes 1.0-rc1, or the puppet module dependency type 1.x which our librarian gem will adequately convert to ~>1.0 instead of failing. Check out the rubygems version policies for more details.

Next post I’ll cover how to use librarian in combination with puppetlabs spec helper to make your puppet testing much easier and consistent, and I’ll cover the topic in my upcoming talk on continuous delivery at ApacheCon Portland in February, plus you’ll find me next monday the 28th at the Los Angeles DevOps meetup, and at DevOps Days LA February 22nd as part of the SCALE conference.