Using Puppet’s metadata.json in Librarian-Puppet and Blacksmith

I have published new versions of librarian-puppet and puppet-blacksmith gems that handle the new Puppet metadata.json format for module data and dependencies.

Puppet Labs logolibrarian-puppet 1.3.1 and 1.0.8 [changelog] include two important changes. Now there is no need to create a Puppetfile if you have a Modulefile or metadata.json, it will use them by default. Of course you can add a Puppetfile to bring in modules from git, a directory, or github tarballs.

The other change is that all the dependencies’ metadata.json files will be parsed now for transitive dependencies, so it works with the latest Puppet Labs modules and those migrated from the old Modulefile format going forward. That also means that the puppet gem is no longer needed if there are no Modulefile present in your tree of dependencies, which was a source of pain for some users.

The 1.0.x branch is kept updated to run in Ruby 1.8 while 1.1+ requires Ruby 1.9 and uses the Puppet Forge API v3.

Puppet Blacksmith, the gem to automate pushing modules to the Puppet Forge was also updated to use metadata.json besides Modulefile in version 2.2+ [changelog].

librarian-puppet 1.1 released with new Puppet Forge support

Just released librarian-puppet version 1.1.0, a version that adds support for the new Puppet Forge v3 API and fixes the issues in Puppet 3.6+ and Puppet Enterprise 3.2+, versions that started using the new v3 API. From 1.1 the ruby requirement is 1.9+ due to the puppet_forge library used.

 

librarian-puppetLibrarian-puppet is a bundler for your puppet infrastructure. You can use librarian-puppet to manage the puppet modules your infrastructure depends on, whether the modules come from the Puppet Forge, Git repositories or a just a path.

  • Librarian-puppet can reuse the dependencies listed in your Modulefile
  • Forge modules can be installed from Puppetlabs Forge or an internal Forge such as Pulp
  • Git modules can be installed from a branch, tag or specific commit, optionally using a path inside the repository
  • Modules can be installed from GitHub using tarballs, without needing Git installed
  • Module dependencies are resolved transitively without needing to list all the modules explicitly

Librarian-puppet manages your modules/ directory for you based on your Puppetfile. Your Puppetfile becomes the authoritative source for what modules you require and at what version, tag or branch.

Changelog

1.1.1

  • Issue #227 Fix Librarian::Puppet::VERSION undefined

1.1.0

  • Issue #210 Use forgeapi.puppetlabs.com and API v3
    • Accesing the v3 API requires Ruby 1.9 due to the puppet_forge library used

1.0.3

  • Issue #223 Cannot bounce Puppetfile.lock! error when Forge modules contain duplicated dependencies

1.0.2

  • Issue #211 Pass the PuppetLabs Forge API v3 endpoint to puppet module when running on Puppet >= 3.6.0
  • Issue #198 Reduce the length of tmp dirs to avoid issues in windows
  • Issue #206 githubtarball call for released versions does not consider pagination
  • Issue #204 Fix regex to detect Forge API v3 url
  • Issue #199 undefined method run! packaging a git source
  • Verify SSL certificates in github calls

1.0.1

  • Issue #190 Pass the PuppetLabs Forge API v3 endpoint to puppet module when running on Puppet Enterprise >= 3.2
  • Issue #196 Fix error in error handling when puppet is not installed

 

Announcing librarian-puppet 1.0.0

librarian-puppetI’m proud to announce the release of librarian-puppet version 1.0.0. It was about time to get to 1.x after more than 200k gem installations. See my previous post about managing Puppet modules to take advantage of its features.

Librarian-puppet is a bundler for your puppet infrastructure. You can use librarian-puppet to manage the puppet modules your infrastructure depends on, whether the modules come from the Puppet Forge, Git repositories or a just a path.

  • Librarian-puppet can reuse the dependencies listed in your Modulefile
  • Forge modules can be installed from Puppetlabs Forge or an internal Forge such as Pulp
  • Git modules can be installed from a branch, tag or specific commit, optionally using a path inside the repository
  • Modules can be installed from GitHub using tarballs, without needing Git installed
  • Module dependencies are resolved transitively without needing to list all the modules explicitly

Librarian-puppet manages your modules/ directory for you based on your Puppetfile. Your Puppetfile becomes the authoritative source for what modules you require and at what version, tag or branch.

Changelog

1.0.0

  • Remove deprecation warning for github_tarball sources, some people are actually using it

0.9.17

0.9.16

  • Issue #181 Should use qualified module names for resolution to work correctly
  • Deprecate github_tarball sources
  • Reduce number of API calls for github_tarball sources

0.9.15

  • Issue #187 Fixed parallel installation issues
  • Issue #185 Sanitize the gem/bundler environment before spawning (ruby 1.9+)

0.9.14

  • Issue #182 Sanitize the environment before spawning (ruby 1.9+)
  • Issue #184 Support transitive dependencies in modules using :path
  • Git dependencies using modulefile syntax make librarian-puppet fail
  • Issue #108 Don’t fail on malformed Modulefile from a git dependency

0.9.13

  • Issue #176 Upgrade to librarian 0.1.2
  • Issue #179 Need to install extra gems just in case we are in ruby 1.8
  • Issue #178 Print a meaningful message if puppet gem can’t be loaded for :git sources

0.9.12

  • Remove extra dependencies from gem added when 0.9.11 was released under ruby 1.8

0.9.11

  • Add modulefile dsl to reuse Modulefile dependencies
  • Consider Puppetfile-dependencies recursively in git-source
  • Support changing tmp, cache and scratch paths
  • librarian-puppet package causes an infinite loop
  • Show a message if no versions are found for a module
  • Make download of tarballs more robust
  • Require open3_backport in ruby 1.8 and install if not present
  • Git dependencies in both Puppetfile and Modulefile cause a Cannot bounce Puppetfile.lock! error
  • Better sort of github tarball versions when there are mixed tags starting with and without ‘v’
  • Fix error if a git module has a dependency without version
  • Fix git dependency with :path attribute
  • Cleaner output when no Puppetfile found
  • Reduce the number of API calls to the Forge
  • Don’t sort versions as strings. Rely on the forge returning them ordered
  • Pass –module_repository to puppet module install to install from other forges
  • Cache forge responses and print an error if returns an invalid response
  • Add a User-Agent header to all requests to the GitHub API
  • Convert puppet version requirements to rubygems, pessimistic and ranges
  • Use librarian gem

0.9.10

  • Catch GitHub API rate limit exceeded
  • Make Librarian::Manifest Semver 2.0.0 compatible

 

New release of librarian puppet

Puppet Labs logoI’ve been helping with the development of librarian-puppet, pushing upstream a lot of fixes we had made in the past and applying long outstanding pull requests in the project in order to get a release out, and finally you can get the (probably) last release before 1.0.0 which should be stable enough for day to day use.

Besides bug fixes probably the best feature is the ability of reusing the Modulefile dependencies by creating the simplest Puppetfile, if you only need modules from the Puppet Forge

forge "http://forge.puppetlabs.com"

modulefile

 

The changelog

0.9.13

  • Issue #176 Upgrade to librarian 0.1.2
  • Issue #179 Need to install extra gems just in case we are in ruby 1.8
  • Issue #178 Print a meaningful message if puppet gem can’t be loaded for :git sources

0.9.12

  • Remove extra dependencies from gem added when 0.9.11 was released under ruby 1.8

0.9.11

  • Add modulefile dsl to reuse Modulefile dependencies
  • Consider Puppetfile-dependencies recursively in git-source
  • Support changing tmp, cache and scratch paths
  • librarian-puppet package causes an infinite loop
  • Show a message if no versions are found for a module
  • Make download of tarballs more robust
  • Require open3_backport in ruby 1.8 and install if not present
  • Git dependencies in both Puppetfile and Modulefile cause a Cannot bounce Puppetfile.lock! error
  • Better sort of github tarball versions when there are mixed tags starting with and without ‘v’
  • Fix error if a git module has a dependency without version
  • Fix git dependency with :path attribute
  • Cleaner output when no Puppetfile found
  • Reduce the number of API calls to the Forge
  • Don’t sort versions as strings. Rely on the forge returning them ordered
  • Pass –module_repository to puppet module install to install from other forges
  • Cache forge responses and print an error if returns an invalid response
  • Add a User-Agent header to all requests to the GitHub API
  • Convert puppet version requirements to rubygems, pessimistic and ranges
  • Use librarian gem

Testing puppet modules

Puppet Labs logoThere are several steps depending on how much involved the tests are, what parts are tested and, of course, how long it takes to run the tests.

For unit testing we use rspec puppet, and we can check that our manifests and modules compile and contain the expected values. It can be used to test that specific types, classes or definitions are in the compiled catalog and that the parameters math the expectations.

Later on we can do some integration testing starting a new VM with Vagrant and checking that there are no errors in the provisioning, as well as checking that some conditions are met.

For rspec-puppet, PuppetLabs has created a project called puppetlabs_spec_helper that let’s us avoid writing a bunch of boilerplate. A missing point though is that it only allows to use modules for testing from git. If you’re already using librarian-puppet (and you should!) you can easily use the same Puppetfile for deploying modules and to test them. Doing otherwise sounds like a bit of useless testing, you could end with different versions in different development machines, CI server, puppet master,… So just add a call to librarian puppet in your rakefile to populate the rspec-puppet fixtures before running the specs.

Unfortunately rspec-puppet doesn’t work with Puppet 3.0.x and  at least Puppet 3.1.0-rc1 is required. It was a bit of a setback when we moved to Puppet 3 and started using hiera, which is proving to be very useful to have simpler manifests and external data injected for our Maestro installations with Puppet from scratch.

You can also use the same Puppetfile to start Vagrant boxes with the exact same version of the modules. We are using Cucumber and Aruba to execute vagrant, provision the VM with puppet and check several things, like open ports, services up,… but that’s a different story :-)

Example

In this puppet-for-java-devs project you will find the bits that showcase all these tools integrated. It includes definition of a 3-tier system with Puppet definitions for a postgresql database, tomcat nodes with a war installed and apache nodes fronting them.

Install all required gems

bundle install

Install all Puppet modules with Puppet Librarian

librarian-puppet install

Run the specs with puppet-rspec

bundle exec rake

Start all the vms with Vagrant

vagrant up

Rakefile

require 'bundler'
Bundler.require(:rake)
require 'rake/clean'

require 'puppetlabs_spec_helper/rake_tasks'

CLEAN.include('modules', 'spec/fixtures/', 'doc')
CLOBBER.include('.tmp', '.librarian')

task :librarian_spec_prep do
 sh "librarian-puppet install"
end
task :spec_prep => :librarian_spec_prep

task :default => [:spec]

Puppetfile for librarian-puppet

forge 'http://forge.puppetlabs.com'

mod 'puppetlabs/java', '0.1.6'
mod 'puppetlabs/apache', '0.4.0'
mod 'inkling/postgresql', '0.2.0'
mod 'puppetlabs/firewall', '0.0.4'
mod 'tomcat', :git => 'https://github.com/carlossg/puppet-tomcat.git', :ref => 'centos'
mod 'maestrodev/maven', '1.x'
mod 'stahnma/epel', '0.0.2'
mod 'maestrodev/avahi', '1.x'
mod 'other', :path => 'mymodules/other'

tomcat_spec.rb with rspec-puppet

require 'spec_helper'

describe 'tomcat1.acme.com' do
  let(:facts) { {:osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => 6.3} }

  it { should contain_class('java').with_distribution /openjdk/ }

  it "configure webapp" do
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT.war')
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT/WEB-INF/lib/postgresql-9.1-901.jdbc4.jar')
  end
end

Vagrantfile

Vagrant::Config.run do |config|
  config.vm.box = "CentOS-6.3-x86_64-minimal"
  config.vm.box_url = "https://dl.dropbox.com/u/7225008/Vagrant/CentOS-6.3-x86_64-minimal.box"

  config.vm.customize ["modifyvm", :id, "--rtcuseutc", "on"] # use UTC clock https://github.com/mitchellh/vagrant/issues/912

  # db server
  config.vm.define :db do |config|
    config.vm.host_name = "db.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "db"] # name for VirtualBox GUI
    config.vm.forward_port 5432, 5432
    config.vm.network :hostonly, "192.168.33.10"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # tomcat server
  config.vm.define :tomcat1 do |config|
    config.vm.host_name = "tomcat1.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "tomcat1"] # name for VirtualBox GUI
    config.vm.forward_port 8080, 8081
    config.vm.network :hostonly, "192.168.33.11"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # web server
  config.vm.define :www do |config|
    config.vm.host_name = "www.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "www"] # name for VirtualBox GUI
    config.vm.forward_port 80, 8080
    config.vm.network :hostonly, "192.168.33.12"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

end

Managing Puppet modules with librarian-puppet

Puppet Labs logoOnce you have your Infrastructure defined as code you need to use coding best practises.

This definitely applies to Puppet code, you have Puppet classes to encapsulate functionality, and modules that group classes together for reuse and redistribution. Modules can be reused from git repositories or from the Puppet Forge, a web archive of tarballed modules in the same fashion as rpm or maven repositories. Modules in the forge are defined by a provider, module name and version, much like Maven groupid, artifactid and version, and a provider has full control over the modules that can be published under its name, with a more open approach, as anybody can upload anything themselves through the forge website, just signup and start uploading them.

But trouble starts when you are using a number of modules from different places, as each module can have its own set of dependencies. How can the dependencies be defined, downloaded and installed automatically as happens with any other package manager?

There are several options actually

  • Git submodules. This is the poor man’s version, where you add the git repositories of other modules as git submodules and use git submodule init and update commands to stay up to date
  • Puppet module tool. Allows the installation of modules from the Puppet forge based on the module name provider/name and version. It will download and unpackage the module in the directory specified
  • Librarian-puppet. It is an extension of ruby’s bundler model to install gems, having a file that defines your dependencies, Gemfile in the case of bundler, Puppetfile in the case of librarian, and a .lock file with the resolved dependencies.

The resolution model is different than those of yum, apt-get or maven, that resolve dependencies at every run on the client, relying on the clients to consistently do that over versions and machines. In the bundler or librarian model, resolution only happens once and it’s saved to the lock file, which is used from there on by the clients until dependencies are changed.

Dependency declaration

The librarian Puppetfile allows defining module dependencies from several sources

Puppet Forge

Using the forge provider and module name, and optionally version it calls puppet module tool to fetch the tarball and extract it in the designed directory

mod "maestrodev/maven", "1.0.0"

Git

Modules can also be cloned from a git repository optionally defining what branch, tag or revision to checkout

mod "maven",
  :git => "https://github.com/maestrodev/puppet-maven.git",
  :ref => 'v1.0.0'

Path folders

Useful mostly for testing or while transitioning to a full dependency model, local paths can be used as modules

mod 'mymodule', :path => './private/mymodule'

Improvements and issues

Librarian offers several improvements over the puppet module tool. Besides being able to bring in modules from different sources, librarian locks the versions of modules used, which is a must in order to consistently reproduce results. Let’s say you define a dependency on module maestrodev/maven > 1.0.0. If you run it now then librarian will lock the version to 1.0.0. If 1.1.0 is released later your dependencies will stay the same. If you try to run the puppet module tool with

puppet module install maestrodev-maven

you would end with 1.0.0 now and  a newer version later on.

There are some problems, and we hit (and fixed!) a lot of them. We have a fork of the original project where we keep updating the maestrodev branch while our pull requests don’t make it to the original repo. Unfortunately some of the patches have been sitting there for too long and we went ahead and created the librarian-puppet-maestrodev gem  that you can use instead of the original one.

Some of the issues you’ll likely find are the mismatch between puppet module version conventions and gem versions. librarian uses the bundler gemfile resolution mechanism which does not accept some versions such as the ones including dashes 1.0-rc1, or the puppet module dependency type 1.x which our librarian gem will adequately convert to ~>1.0 instead of failing. Check out the rubygems version policies for more details.

Next post I’ll cover how to use librarian in combination with puppetlabs spec helper to make your puppet testing much easier and consistent, and I’ll cover the topic in my upcoming talk on continuous delivery at ApacheCon Portland in February, plus you’ll find me next monday the 28th at the Los Angeles DevOps meetup, and at DevOps Days LA February 22nd as part of the SCALE conference.