New release of librarian puppet

Puppet Labs logoI’ve been helping with the development of librarian-puppet, pushing upstream a lot of fixes we had made in the past and applying long outstanding pull requests in the project in order to get a release out, and finally you can get the (probably) last release before 1.0.0 which should be stable enough for day to day use.

Besides bug fixes probably the best feature is the ability of reusing the Modulefile dependencies by creating the simplest Puppetfile, if you only need modules from the Puppet Forge

forge "http://forge.puppetlabs.com"

modulefile

 

The changelog

0.9.13

  • Issue #176 Upgrade to librarian 0.1.2
  • Issue #179 Need to install extra gems just in case we are in ruby 1.8
  • Issue #178 Print a meaningful message if puppet gem can’t be loaded for :git sources

0.9.12

  • Remove extra dependencies from gem added when 0.9.11 was released under ruby 1.8

0.9.11

  • Add modulefile dsl to reuse Modulefile dependencies
  • Consider Puppetfile-dependencies recursively in git-source
  • Support changing tmp, cache and scratch paths
  • librarian-puppet package causes an infinite loop
  • Show a message if no versions are found for a module
  • Make download of tarballs more robust
  • Require open3_backport in ruby 1.8 and install if not present
  • Git dependencies in both Puppetfile and Modulefile cause a Cannot bounce Puppetfile.lock! error
  • Better sort of github tarball versions when there are mixed tags starting with and without ‘v’
  • Fix error if a git module has a dependency without version
  • Fix git dependency with :path attribute
  • Cleaner output when no Puppetfile found
  • Reduce the number of API calls to the Forge
  • Don’t sort versions as strings. Rely on the forge returning them ordered
  • Pass –module_repository to puppet module install to install from other forges
  • Cache forge responses and print an error if returns an invalid response
  • Add a User-Agent header to all requests to the GitHub API
  • Convert puppet version requirements to rubygems, pessimistic and ranges
  • Use librarian gem

Installing Puppet 3 in a BeagleBone or Raspberry Pi

Puppet Labs logoI am a proud owner of a NinjaBlocks device that I use to control my home (blinds, hot water, heater, presence detection,…), welcome to the Internet of Things! But that’s a story for next posts, the important thing is that the device is actually a BeagleBone board running Ubuntu connected to an Arduino board.

raspberry logo

I thought, what would better than managing the NinjaBlock ubuntu with Puppet? there are a number of files and services I added there and it’d be nice to have puppet installed. But the official Ubuntu packages only offer puppet 2.7.11, so I installed the PuppetLabs repository package and tried to install Puppet 3, failing miserably because there is no Facter build for armhf platform,  the same one used in Raspbian.


wget https://apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb
sudo apt-get update
sudo apt-get install puppet

and got this

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
puppet : Depends: puppet-common (= 3.4.2-1puppetlabs1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

When I tried to install all the required packages


sudo apt-get install puppet puppet-common facter

then I found the actual error

The following packages have unmet dependencies:
facter : Depends: dmidecode but it is not installable
E: Unable to correct problems, you have held broken packages.

Investigating a bit I found that facter has a dependency on dmidecode that should be optional, as dmidecode is not available for arm platform and is not really needed for facter.

The solution? Rebuild the facter package removing that dependency, easily done with this script. When vi opens just delete the dmidecode dependency and you will get the fixed facter_1.6.18-1puppetlabs1_all.modified.deb

curl -O http://apt.puppetlabs.com/pool/precise/main/f/facter/facter_1.6.18-1puppetlabs1_all.deb
curl -O curl -O https://gist.github.com/carlossg/8578202/raw/70689b1b74517cc4b0743e54d84dae8375503159/videbcontrol.sh
bash videbcontrol.sh facter_1.6.18-1puppetlabs1_all.deb
# remove dependency to dmidecode in the editor that opens
sudo dpkg -i facter_1.6.18-1puppetlabs1_all.modified.deb
# If you want to use ruby 1.9 instead of 1.8
sudo apt-get install libaugeas-ruby1.9.1 ruby1.9.1

# puppet may not install if all the dependencies are not listed
sudo apt-get install puppet puppet-common
# mark dependencies as automatically installed so they are removed when removing puppet
sudo apt-mark auto facter libaugeas-ruby1.9.1 puppet-common

Installing RVM and multiple ruby versions with Puppet

rvm_logoWith the latest version of the Puppet RVM module it is even easier to install multiple versions of Ruby


# install rubies from binaries
Rvm_system_ruby {
  ensure     => present,
  build_opts => ['--binary'],
}

# ensure rvm doesn't timeout finding binary rubies
# the umask line is the default content when installing rvm if file does not exist
file { '/etc/rvmrc':
  content => 'umask u=rwx,g=rwx,o=rx
                     export rvm_max_time_flag=20',
  mode    => '0664',
  before  => Class['rvm'],
}

class { 'rvm': }
rvm::system_user { 'vagrant': }
rvm_system_ruby {
  'ruby-1.9.3':
    default_use => true;
  'ruby-2.0.0':;
  'jruby':;
}

Hiera can also be used to define what rubies to install, making the Puppet code even less verbose

...
class { 'rvm': }
# rvm::system_user no longer needed
# rvm_system_ruby no longer needed

The equivalent hiera yaml configuration to the previous example

rvm::system_rubies:
  '1.9':
    default_use: true
  '2.0': {}
  'jruby-1.7': {}

rvm::system_users:
  - vagrant

Continuous Delivery with Maven, Puppet and Tomcat – Video from ApacheCon NA 2013

Apachecon NA 2013A little bit late but finally the video from my session at ApacheCon Portland is available. That was the first version of the talk that I just gave at Agile testing Days which unfortunately was not recorded.

Description
Continuous Integration, with Apache Continuum or Jenkins, can be extended to fully manage deployments and production environments, running in Tomcat for instance, in a full Continuous Delivery cycle using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations.

Abstract
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,… in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Apache Continuum or Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,… and easily make changes that propagate across all nodes.

Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.

We will show how to install and manage Puppet nodes with JDK, multiple Tomcat instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.

Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days

agiletdThis week I’m in Postdam/Berlin giving a talk Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days. Showing examples of using Puppet, Vagrant and other tools to implement a source code to production continuous delivery cycle.

Slides are up in SlideShare, and source code is available at GitHub.

Extend Continuous Integration to automatically test your infrastructure.

Continuous Integration can be extended to test deployments and production environments, in a Continuous Delivery cycle, using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations, and test the infrastructure the same way continuous integration tools do with developers’ code.

Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services, … in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Using Vagrant, a command line automation layer for VirtualBox, we can easily spin off virtual machines with the same configuration as production servers, run our test suite, and tear them down afterwards.

We will show how to set up automated testing of an application and associated infrastructure and configurations, creating on demand virtual machines for testing, as part of your continuous integration process.

Testing Puppet and Hiera

Puppet Labs logoAt MaestroDev we have been using Puppet 3 for a quite some time now, and one of the main reasons to upgrade from Puppet 2.x was the ability of using Hiera as a data backend for all the variables that customize the different vms. We don’t have a lot of machines but pretty much all of them have some difference so Hiera allows us to have the same manifests and modules apply to all the machines by just using different parameters in each server.

But testing Hiera is not that simple. With rspec-puppet you can test each class by passing parameters, but how can you test a class that calls another class and so on, and at some point there you need to inject a parameter?

Well, this is possible with the hiera-puppet-helper gem you can stub the hiera data backend

source 'https://rubygems.org'

group :rake do
gem 'puppet'
gem 'rspec-puppet'
gem 'hiera-puppet-helper'
gem 'rake'
gem 'puppetlabs_spec_helper'
end
require 'puppetlabs_spec_helper/module_spec_helper'
require 'hiera-puppet-helper/rspec'
require 'hiera'
require 'puppet/indirector/hiera'

# config hiera to work with let(:hiera_data)
def hiera_stub
  config = Hiera::Config.load(hiera_config)
  config[:logger] = 'puppet'
  Hiera.new(:config => config)
end

RSpec.configure do |c|
  c.mock_framework = :rspec

  c.before(:each) do
    Puppet::Indirector::Hiera.stub(:hiera => hiera_stub)
  end

end

And then you can use let(:hiera_data) to inject any parameters automatically into the puppet classes from your rspec tests.

require 'spec_helper'

describe 'mymodule::myclass' do

let(:hiera_data) {{
  'mymodule::myclass::myparam' => 'myvalue'
}}

it { should contain_class('mymodule::myclass').with_myparam('myvalue') }

Check out a full module using hiera-puppet-helper at maestro_nodes.

PuppetConf video: How to Develop Puppet Modules

How to Develop Puppet Modules. From Source to the Forge With Zero Clicks (slides)

Puppet Modules are a great way to reuse code, share your development with other people and take advantage of the hundreds of modules already available in the community. But how to create, test and publish them as easily as possible? now that infrastructure is defined as code, we need to use development best practices to build, test, deploy and use Puppet modules themselves.

Three steps for a fully automated process

  • Continuous Integration of Puppet Modules
  • Automatic release and upload to the Puppet Forge
  • Deploy to Puppet master

More about PuppetConf in my previous entry.