Security Testing Using Infrastructure-As-Code

Agile RecordArticle originally published at Agile Record magazine Issue #17 Security Testing in an Agile Environment. Can be downloaded for free as a PDF.

Security Testing Using Infrastructure-As-Code

Infrastructure-As-Code means that infrastructure should be treated as code – a really powerful concept. Server configuration, packages installed, relationships with other servers, etc. should be modeled with code to be automated and have a predictable outcome, removing manual steps prone to errors. That doesn’t sound bad, does it?

The goal is to automate all the infrastructure tasks programmatically. In an ideal world you should be able to start new servers, configure them, and, more importantly, be able to repeat it over and over again, in a reproducible way, automatically, by using tools and APIs.

Have you ever had to upgrade a server without knowing whether the upgrade was going to succeed or not for your application? Are the security updates going to affect your application? There are so many system factors that can indirectly cause a failure in your application, such as different kernel versions, distributions, or packages.

When you have a decent set of integration tests it is not that hard to make changes to your infrastructure with that safety net. There are a number of tools designed to make your life easier, so there is no need to tinker with bash scripts or manual steps prone to error.

We can find three groups of tools:

  • Provisioning tools, like Puppet or Chef, manage the configuration of servers with packages, services, config files, etc. in a reproducible way and over hundreds of machines.
  • Virtual Machine automation tools, like Vagrant, enable new virtual machines to be started easily in different environments, from virtual machines in VirtualBox or VMware to cloud providers such as Amazon AWS or Rackspace, and then provision them with Puppet or Chef.
  • Testing tools, like rspec, Cucumber, or Selenium, enable unit and integration tests to be written that verify that the server is in a good state continuously as part of your continuous integration process.

Vagrant

Learning Puppet can be a tedious task, such as getting up the different pieces (master, agents), writing your first manifests, etc. A good way to start is to use Vagrant, which started as an Oracle VirtualBox command line automation tool, and allows you to create new VMs locally or on cloud providers and provision them with Puppet and Chef easily.

Vagrant projects are composed of base boxes, specifically configured for Vagrant with Puppet/Chef, vagrant username and password, and any customizations you may want to add, plus the configuration to apply to those base boxes defined with Puppet or Chef. That way we can have several projects sharing the same base boxes where the Puppet/Chef definitions are different. For instance, a database VM and a web server VM can both use the same base box, i.e. a CentOS 6 minimal server, and just have different Puppet manifests. When Vagrant starts them up it will apply the specific configuration. That also allows you to share boxes and configuration files across teams. For instance, one base box with the Linux flavor can be used in a team, and in source control we can have just the Puppet manifests to apply for the different configurations that anybody from Operations to Developers can use. If a problem arises in production, a developer can quickly instantiate a equivalent environment using the Vagrant and Puppet configuration, making a different environment’s issues easy to reproduce.

There is a list of available VMs or base boxes ready to use with Vagrant at www.vagrantbox.es, but you can build your own and share it anywhere. For VirtualBox they are just (big) VM files that can be easily built using VeeWee (https://github.com/jedi4ever/veewee) or by changing a base box and rebundling it with Packer (http://www.packer.io).

Usage

Once you have installed Vagrant (http://docs.vagrantup.com/v2/installation/index.html) and VirtualBox (https://www.virtualbox.org/) you can create a new project.

Vagrant init will create a sample Vagrantfile, the project definition file that can be customized.

$ vagrant init myproject

Then in the Vagrantfile you can change the default box settings and add basic Puppet provisioning.

config.vm.box = "CentOS-6.4-x86_64-minimal"
config.vm.box_url = "https://repo.maestrodev.com/archiva/repository/public-releases/com/maestrodev/vagrant/CentOS/6.4/CentOS-6.4-x86_64-minimal.box"

# create a virtual network so we can access the vm by ip
config.vm.network "private_network", ip: "192.168.33.13"
config.vm.hostname = "qa.acme.local"
config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "manifests"
  puppet.manifest_file = "site.pp"
  puppet.module_path = "modules"
  end

In manifests/site.pp you can try any puppet code, i.e. create a file

node 'qa.acme.local' {
  file { '/root/secret':
  mode => '0600',
  owner => 'root',
  content => 'secret file, for root eyes only',
  }
}

Vagrant up will download the box the first time, start the VM, and apply the configuration defined in Puppet.

$ vagrant up

vagrant ssh will open a shell into the box. Under the hood, vagrant is redirecting a host port to vagrant box 22.

$ vagrant ssh

If you make any changes to the Puppet manifests you can rerun the provisioning step.

$ vagrant provision

The vm can be suspended and resumed at any time

$ vagrant suspend
$ vagrant resume

and later on destroyed, which will delete all the VM files.

$ vagrant destroy

And then we can start again from scratch with vagrant up getting a completely new vm where we can make any mistakes!

Puppet

In Puppet we can configure any aspect of a server: packages, files, permissions, services, etc. You have seen how to create a file, now let’s see an example of configuring Apache httpd server and the Linux iptables firewall to open a port.

First we need the Puppet modules to manage httpd and the firewall rules to avoid writing all the bits and pieces ourselves. Modules are Puppet reusable components that you can find at the Puppet Forge (http://forge.puppetlabs.com/) or typically in GitHub. To install these two modules into the vm, run the following commands that will download the modules and install them in the /etc/puppet/modules directory.

vagrant ssh -c "sudo puppet module install --version 0.9.0 puppetlabs/apache"
vagrant ssh -c "sudo puppet module install --version 0.4.2 puppetlabs/firewall"

You can find more information about the Apache (http://forge.puppetlabs.com/puppetlabs/apache/0.9.0) and the Firewall (http://forge.puppetlabs.com/puppetlabs/firewall/0.4.2) modules in their Forge pages. We are just going to add some simple examples to the manifests/site.pp to install the Apache server with a virtual host that will listen in port 80.

node 'qa.acme.local' {

  class { 'apache': }

  # create a virtualhost

  apache::vhost { "${::hostname}.local":
    port => 80,
    docroot => '/var/www',
  }
  }

Now if you try to access this server in port 80 you will not be able to, as iptables is configured by default to block all incoming connections. Try accessing http://192.168.33.13 (the ip we configured previously in the Vagrantfile for the private virtual network) and see for yourself.

To open the firewall, we need to open the port explicitly in the manifests/site.pp by adding

firewall { '100 allow apache':
  proto => 'tcp',
  port => '80',
  action => 'accept',
  }

and running vagrant provision again. Now you should see Apache’s default page in http://192.168.33.13.

So far we have created a virtual machine where the apache server is automatically installed and the firewall open. You could start from scratch at any time by running vagrant destroy and vagrant up again.

Testing

Let’s write some tests to ensure that everything is working as expected. We are going to use Ruby as the language of choice.

Unit testing with rspec-puppet

rspec-puppet (http://rspec-puppet.com/) is a rspec extension that allows to easily unit test Puppet manifests.

Create a spec/spec_helper.rb file to add some shared config for all the specs

require 'rspec-puppet'

RSpec.configure do |c|
  c.module_path = 'modules'
  c.manifest_dir = 'manifests'
  end

and we can start creating unit tests for the host that we defined in Puppet.

# spec/hosts/qa_spec.rb

require 'spec_helper'

describe 'qa.acme.local' do

  # test that the httpd package is installed

  it { should contain_package('httpd') }

  # test that there is a firewall rule set to 'accept'

  it { should contain_firewall('100 allow apache').with_action('accept') }

  # ensure that there is only one firewall definition

  it { should have_firewall_resource_count(1) }

  end

After installing rspec-puppet gem install rspec-puppet, you can run rspec to execute the tests.

...

Finished in 1.4 seconds

3 examples, 0 failures

Success!

Integration testing with Cucumber

Unit testing is fast and can catch a lot of errors quickly, but how can we check that the machine is actually configured as we expected?

Let’s use Cucumber (http://cukes.info/), a BDD tool, to create an integration test that checks whether a specific port is open in the virtual machine we started.

Create a features/smoke_tests.feature file with:

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port
Then the "apache" service should be listening on port "80"

Install Cucumber gem install cucumber and run cucumber. The first run will output a message saying that the step definition has not been created yet.

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port # features/smoke_tests.feature:4
Then the "apache" service should be listening on port "80" # features/smoke_tests.feature:5

1 scenario (1 undefined)

1 step (1 undefined)

0m0.001s

You can implement step definitions for undefined steps with these snippets:

Then(/^the "(.*?)" service should be listening on port "(.*?)"$/) do |arg1, arg2|
  pending # express the regexp above with the code you wish you had
  end

So let’s create a features/step_definitions/tcp_ip_steps.rb file that implements our service should be listening on port step by opening a TCP socket.

Then /^the "(.*?)" service should be listening on port "(.*?)"$/ do |service, port|
  host = URI.parse(ENV['URL']).host
  begin
    s = TCPSocket.new(host, port)
    s.close
    rescue Exception => error
    raise("#{service} is not listening at #{host} on port #{port}")
  end
  end

And rerun Cucumber, this time using an environment variable URL to specify where the machine is running, as used in the step definition URL=http://192.168.33.13 cucumber.

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port # features/smoke_tests.feature:4
Then the "apache" service should be listening on port "80" # features/step_definitions/tcp_ip_steps.rb:1

1 scenario (1 passed)

1 step (1 passed)

0m0.003s

Success! The port is actually open in the virtual machine.

Wash, rinse, repeat

This was a small example of what can be achieved using Infrastructure-As-Code and automation tools such as Puppet and Vagrant combined with standard testing tools like rspec or Cucumber. When a continuous integration tool like Jenkins is thrown into the mix to run these tests continuously, the result is an automatic end-to-end solution that tests systems as any other code, avoiding regressions and enabling Continuous Delivery (http://blog.csanchez.org/2013/11/12/continuous-delivery-with-maven-puppet-and-tomcat-video-from-apachecon-na-2013/) – automation all the way from source to production.

A more detailed example can be found in my continuous-delivery project at GitHub (https://github.com/carlossg/continuous-delivery).

New release of librarian puppet

Puppet Labs logoI’ve been helping with the development of librarian-puppet, pushing upstream a lot of fixes we had made in the past and applying long outstanding pull requests in the project in order to get a release out, and finally you can get the (probably) last release before 1.0.0 which should be stable enough for day to day use.

Besides bug fixes probably the best feature is the ability of reusing the Modulefile dependencies by creating the simplest Puppetfile, if you only need modules from the Puppet Forge

forge "http://forge.puppetlabs.com"

modulefile

 

The changelog

0.9.13

  • Issue #176 Upgrade to librarian 0.1.2
  • Issue #179 Need to install extra gems just in case we are in ruby 1.8
  • Issue #178 Print a meaningful message if puppet gem can’t be loaded for :git sources

0.9.12

  • Remove extra dependencies from gem added when 0.9.11 was released under ruby 1.8

0.9.11

  • Add modulefile dsl to reuse Modulefile dependencies
  • Consider Puppetfile-dependencies recursively in git-source
  • Support changing tmp, cache and scratch paths
  • librarian-puppet package causes an infinite loop
  • Show a message if no versions are found for a module
  • Make download of tarballs more robust
  • Require open3_backport in ruby 1.8 and install if not present
  • Git dependencies in both Puppetfile and Modulefile cause a Cannot bounce Puppetfile.lock! error
  • Better sort of github tarball versions when there are mixed tags starting with and without ‘v’
  • Fix error if a git module has a dependency without version
  • Fix git dependency with :path attribute
  • Cleaner output when no Puppetfile found
  • Reduce the number of API calls to the Forge
  • Don’t sort versions as strings. Rely on the forge returning them ordered
  • Pass –module_repository to puppet module install to install from other forges
  • Cache forge responses and print an error if returns an invalid response
  • Add a User-Agent header to all requests to the GitHub API
  • Convert puppet version requirements to rubygems, pessimistic and ranges
  • Use librarian gem

Installing RVM and multiple ruby versions with Puppet

rvm_logoWith the latest version of the Puppet RVM module it is even easier to install multiple versions of Ruby


# install rubies from binaries
Rvm_system_ruby {
  ensure     => present,
  build_opts => ['--binary'],
}

# ensure rvm doesn't timeout finding binary rubies
# the umask line is the default content when installing rvm if file does not exist
file { '/etc/rvmrc':
  content => 'umask u=rwx,g=rwx,o=rx
                     export rvm_max_time_flag=20',
  mode    => '0664',
  before  => Class['rvm'],
}

class { 'rvm': }
rvm::system_user { 'vagrant': }
rvm_system_ruby {
  'ruby-1.9.3':
    default_use => true;
  'ruby-2.0.0':;
  'jruby':;
}

Hiera can also be used to define what rubies to install, making the Puppet code even less verbose

...
class { 'rvm': }
# rvm::system_user no longer needed
# rvm_system_ruby no longer needed

The equivalent hiera yaml configuration to the previous example

rvm::system_rubies:
  '1.9':
    default_use: true
  '2.0': {}
  'jruby-1.7': {}

rvm::system_users:
  - vagrant

Continuous Delivery with Maven, Puppet and Tomcat – Video from ApacheCon NA 2013

Apachecon NA 2013A little bit late but finally the video from my session at ApacheCon Portland is available. That was the first version of the talk that I just gave at Agile testing Days which unfortunately was not recorded.

Description
Continuous Integration, with Apache Continuum or Jenkins, can be extended to fully manage deployments and production environments, running in Tomcat for instance, in a full Continuous Delivery cycle using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations.

Abstract
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,… in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Apache Continuum or Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,… and easily make changes that propagate across all nodes.

Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.

We will show how to install and manage Puppet nodes with JDK, multiple Tomcat instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.

Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days

agiletdThis week I’m in Postdam/Berlin giving a talk Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days. Showing examples of using Puppet, Vagrant and other tools to implement a source code to production continuous delivery cycle.

Slides are up in SlideShare, and source code is available at GitHub.

Extend Continuous Integration to automatically test your infrastructure.

Continuous Integration can be extended to test deployments and production environments, in a Continuous Delivery cycle, using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations, and test the infrastructure the same way continuous integration tools do with developers’ code.

Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services, … in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Using Vagrant, a command line automation layer for VirtualBox, we can easily spin off virtual machines with the same configuration as production servers, run our test suite, and tear them down afterwards.

We will show how to set up automated testing of an application and associated infrastructure and configurations, creating on demand virtual machines for testing, as part of your continuous integration process.

Testing Puppet and Hiera

Puppet Labs logoAt MaestroDev we have been using Puppet 3 for a quite some time now, and one of the main reasons to upgrade from Puppet 2.x was the ability of using Hiera as a data backend for all the variables that customize the different vms. We don’t have a lot of machines but pretty much all of them have some difference so Hiera allows us to have the same manifests and modules apply to all the machines by just using different parameters in each server.

But testing Hiera is not that simple. With rspec-puppet you can test each class by passing parameters, but how can you test a class that calls another class and so on, and at some point there you need to inject a parameter?

Well, this is possible with the hiera-puppet-helper gem you can stub the hiera data backend

source 'https://rubygems.org'

group :rake do
gem 'puppet'
gem 'rspec-puppet'
gem 'hiera-puppet-helper'
gem 'rake'
gem 'puppetlabs_spec_helper'
end
require 'puppetlabs_spec_helper/module_spec_helper'
require 'hiera-puppet-helper/rspec'
require 'hiera'
require 'puppet/indirector/hiera'

# config hiera to work with let(:hiera_data)
def hiera_stub
  config = Hiera::Config.load(hiera_config)
  config[:logger] = 'puppet'
  Hiera.new(:config => config)
end

RSpec.configure do |c|
  c.mock_framework = :rspec

  c.before(:each) do
    Puppet::Indirector::Hiera.stub(:hiera => hiera_stub)
  end

end

And then you can use let(:hiera_data) to inject any parameters automatically into the puppet classes from your rspec tests.

require 'spec_helper'

describe 'mymodule::myclass' do

let(:hiera_data) {{
  'mymodule::myclass::myparam' => 'myvalue'
}}

it { should contain_class('mymodule::myclass').with_myparam('myvalue') }

Check out a full module using hiera-puppet-helper at maestro_nodes.

PuppetConf video: How to Develop Puppet Modules

How to Develop Puppet Modules. From Source to the Forge With Zero Clicks (slides)

Puppet Modules are a great way to reuse code, share your development with other people and take advantage of the hundreds of modules already available in the community. But how to create, test and publish them as easily as possible? now that infrastructure is defined as code, we need to use development best practices to build, test, deploy and use Puppet modules themselves.

Three steps for a fully automated process

  • Continuous Integration of Puppet Modules
  • Automatic release and upload to the Puppet Forge
  • Deploy to Puppet master

More about PuppetConf in my previous entry.

PuppetConf recap: How to Develop Puppet Modules. From Source to the Forge With Zero Clicks

Puppet Labs logoLast week I was at PuppetConf in San Francisco, speaking about automating the build-test-release-deploy lifecycle for Puppet modules and how we use them. Check out the PuppetLabs blog The Puppet Forge: Modules From Design to Deployment  and see below the original post at MaestroDev’s blog and some (quite unrelated) pictures I took. More conference pictures in the official Flickr account.

Last week PuppetConf took place at the Fairmont in San Francisco, gathering Puppet users and enthusiasts from all over the world for five days of training, development and sessions.

MaestroDev was present at the event as we are heavy Puppet users, and contributors! We are currently the 3rd most frequent contributor to Puppet modules in the Puppet Forge and publish 30 modules, of the 50 that we use on a day to day basis.

Our architect Carlos Sanchez gave a presentation on How to Develop Puppet Modules: From Source to the Forge With Zero Clicks, showing how to apply an automatic build-test-release-deploy cycle for Puppet Modules, because, when you use Infrastructure-as-Code you have to apply the development best practices to your infrastructure. In a demo, he showed how we use Maestro to automatically build, test, release and deploy our Puppet modules to the Puppet Forge, triggered on each commit for a truly Continuous Delivery experience.


And, not only do we use Puppet to provide module deployment to the Forge, we also integrate Maestro with Puppet and Puppet Enterprise to automate Puppet updates across your Puppet agents.  When you put these capabilities together with other development automation capabilities you achieve Continuous Delivery for both your applications and infrastructure.

Imagine you are building or releasing the latest version of your software and need to propagate an update through all the Puppet agents running, as well as any updates to config files or packages managed by Puppet. Instead of waiting for the next Puppet run to happen, Maestro can automatically deploy the Puppet manifests and modules to the Puppet master after being tested, and propagate those changes as well as the latest application built to all or a subset of the servers running with Puppet agents.

Thanks to all the Puppet Labs guys for a great event, and especially to Ryan Coleman for the work he is doing on the Forge and his help getting these Maestro integrations working.  We look forward to seeing everyone at the next event!

Testing puppet modules

Puppet Labs logoThere are several steps depending on how much involved the tests are, what parts are tested and, of course, how long it takes to run the tests.

For unit testing we use rspec puppet, and we can check that our manifests and modules compile and contain the expected values. It can be used to test that specific types, classes or definitions are in the compiled catalog and that the parameters math the expectations.

Later on we can do some integration testing starting a new VM with Vagrant and checking that there are no errors in the provisioning, as well as checking that some conditions are met.

For rspec-puppet, PuppetLabs has created a project called puppetlabs_spec_helper that let’s us avoid writing a bunch of boilerplate. A missing point though is that it only allows to use modules for testing from git. If you’re already using librarian-puppet (and you should!) you can easily use the same Puppetfile for deploying modules and to test them. Doing otherwise sounds like a bit of useless testing, you could end with different versions in different development machines, CI server, puppet master,… So just add a call to librarian puppet in your rakefile to populate the rspec-puppet fixtures before running the specs.

Unfortunately rspec-puppet doesn’t work with Puppet 3.0.x and  at least Puppet 3.1.0-rc1 is required. It was a bit of a setback when we moved to Puppet 3 and started using hiera, which is proving to be very useful to have simpler manifests and external data injected for our Maestro installations with Puppet from scratch.

You can also use the same Puppetfile to start Vagrant boxes with the exact same version of the modules. We are using Cucumber and Aruba to execute vagrant, provision the VM with puppet and check several things, like open ports, services up,… but that’s a different story :-)

Example

In this puppet-for-java-devs project you will find the bits that showcase all these tools integrated. It includes definition of a 3-tier system with Puppet definitions for a postgresql database, tomcat nodes with a war installed and apache nodes fronting them.

Install all required gems

bundle install

Install all Puppet modules with Puppet Librarian

librarian-puppet install

Run the specs with puppet-rspec

bundle exec rake

Start all the vms with Vagrant

vagrant up

Rakefile

require 'bundler'
Bundler.require(:rake)
require 'rake/clean'

require 'puppetlabs_spec_helper/rake_tasks'

CLEAN.include('modules', 'spec/fixtures/', 'doc')
CLOBBER.include('.tmp', '.librarian')

task :librarian_spec_prep do
 sh "librarian-puppet install"
end
task :spec_prep => :librarian_spec_prep

task :default => [:spec]

Puppetfile for librarian-puppet

forge 'http://forge.puppetlabs.com'

mod 'puppetlabs/java', '0.1.6'
mod 'puppetlabs/apache', '0.4.0'
mod 'inkling/postgresql', '0.2.0'
mod 'puppetlabs/firewall', '0.0.4'
mod 'tomcat', :git => 'https://github.com/carlossg/puppet-tomcat.git', :ref => 'centos'
mod 'maestrodev/maven', '1.x'
mod 'stahnma/epel', '0.0.2'
mod 'maestrodev/avahi', '1.x'
mod 'other', :path => 'mymodules/other'

tomcat_spec.rb with rspec-puppet

require 'spec_helper'

describe 'tomcat1.acme.com' do
  let(:facts) { {:osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => 6.3} }

  it { should contain_class('java').with_distribution /openjdk/ }

  it "configure webapp" do
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT.war')
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT/WEB-INF/lib/postgresql-9.1-901.jdbc4.jar')
  end
end

Vagrantfile

Vagrant::Config.run do |config|
  config.vm.box = "CentOS-6.3-x86_64-minimal"
  config.vm.box_url = "https://dl.dropbox.com/u/7225008/Vagrant/CentOS-6.3-x86_64-minimal.box"

  config.vm.customize ["modifyvm", :id, "--rtcuseutc", "on"] # use UTC clock https://github.com/mitchellh/vagrant/issues/912

  # db server
  config.vm.define :db do |config|
    config.vm.host_name = "db.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "db"] # name for VirtualBox GUI
    config.vm.forward_port 5432, 5432
    config.vm.network :hostonly, "192.168.33.10"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # tomcat server
  config.vm.define :tomcat1 do |config|
    config.vm.host_name = "tomcat1.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "tomcat1"] # name for VirtualBox GUI
    config.vm.forward_port 8080, 8081
    config.vm.network :hostonly, "192.168.33.11"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # web server
  config.vm.define :www do |config|
    config.vm.host_name = "www.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "www"] # name for VirtualBox GUI
    config.vm.forward_port 80, 8080
    config.vm.network :hostonly, "192.168.33.12"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

end

Puppet Module of the Week: maestrodev/maven – Maven repository artifact downloads

This is a guest post I wrote in the Puppetlabs blog for their Module of the Week program about the MaestroDev/maven module we created.

Module of the Week: maestrodev/maven – Maven repository artifact downloads

Purpose Manage Apache Maven installation and download artifacts from Maven repositories
Module maestrodev/maven
Puppet Version 2.7+
Platforms RHEL5, RHEL6

The maven module allows Puppet users to install and configure Apache Maven, the build and project management tool, as well as easily use dependencies from Maven repositories.

If you use Maven repositories to store the artifacts resulting from your development process, whether you use Maven, Ivy, Gradle or any other tool capable of pushing builds to Maven repositories, this module defines a new maven type that will let you deploy those artifacts into any Puppet managed server. For instance, you can deploy WAR files directly from your Maven repository by just using their groupId, artifactId and version, bridging development and provisioning without any extra steps or packaging like RPMs or debs.

The maven type allows you to easily provision servers during development by using SNAPSHOT versions—using the latest build for provisioning. Together with a CI tool, this enables you to always keep your development servers up to date.

In this first version, this module supports

  • Installing Apache Maven
  • Configuring Maven settings.xml for repository configuration
  • Configuring Maven environment variables
  • Downloading artifacts from Maven repositories

Installing the module

Complexity Easy
Installation Time 2 minutes

Installing the Maven module is as simple as using the Puppet module tool, available in Puppet 2.7.14+ and Puppet Enterprise 2.5+, and also available as a RubyGem:

$ puppet module install maestrodev-maven
Preparing to install into /etc/puppet/modules ...
Downloading from http://forge.puppetlabs.com ...
Installing -- do not interrupt ...
/etc/puppet/modules
└─┬ maestrodev-maven (v0.0.1)
  └── maestrodev-wget (v0.0.1)

Alternatively, you can install the Maven module manually:

$ cd /etc/puppet/modules/

$ wget http://forge.puppetlabs.com/system/releases/m/maestrodev/maestrodev-maven-0.0.1.tar.gz

$ tar zxvf maestrodev-maven-0.0.1.tar.gz && rm maestrodev-maven-0.0.1.tar.gz
$ mv maestrodev-maven-0.0.1 maven
$ wget http://forge.puppetlabs.com/system/releases/m/maestrodev/maestrodev-wget-0.0.1.tar.gz
$ tar zxvf maestrodev-wget-0.0.1.tar.gz && rm maestrodev-wget-0.0.1.tar.gz
$ mv maestrodev-wget-0.0.1 wget

Resource Overview

CLASSES

maven class

This class installs Apache Maven with a default version of 2.2.1

maven::maven class

Installs Apache Maven, allowing you to specify the version of Maven you wish to install

DEFINITIONS

maven::environment

The definition allows us to configure Apache Maven environment variables on a per-user basis.

maven::settings

Configures $HOME/.m2/settings.xml per user with repositories, mirrors, credentials and properties.

TYPES

maven

This new type lets us download files from remote Maven repositories. Maven must be previously installed.

Testing the module

The module includes some Puppet rspec tests that use the puppetlabs_spec_helper, so it’s simple to implement, and all the fixtures will be automatically downloaded and tests run.

There is a Gemfile included to install all the dependent gems, so after running

$ bundle install

The tests can be executed with

$ bundle exec rake spec

Configuring the module

Complexity Easy
Installation Time 5 minutes

To install Maven there are two options, a simple one to install the default version (2.2.1):

include maven

or a slightly more complex option that customizes the version:

class { "maven::maven":
  version => "3.0.4"
}

Maven will be downloaded by default from the main Apache archive location. It can be configured to be downloaded from a different repository, like one in the local network, by using this repository syntax used throughout the module.

$repo = {
  id       => "myrepo",
  username => "myuser",
  password => "mypassword",
  url      => "http://repo.acme.com",
  mirrorof => "external:*" # if you want to use the repo as a mirror, see maven::settings below
}

class { "maven::maven":
  version => "3.0.4",
  repo    => $repo
}

Once you have Maven installed you can configure the Maven settings.xml for different users, override the mirrors, servers, localRepository, active properties and default repository. It is particularly useful to force Maven to use a repository in the internal network for faster downloads. These settings are used by both command line Maven and the maven puppet type.

We are using hashes to be able to reuse repository definitions, without copy and paste, like the $repo definition above.

# Create a settings.xml with the repo credentials
maven::settings { 'maven' :
  mirrors             => [$central], # mirrors entry in settings.xml, uses id, url, mirrorof from the hash passed
  servers             => [$central], # servers entry in settings.xml, uses id, username, password from the hash passed
  user                => 'maven',
  default_repo_config => {
    url       => $repo['url],
    snapshots => {
      enabled      => 'true',
      updatePolicy => 'always'
    },
    releases  => {
      enabled      => 'true',
      updatePolicy => 'always'
    }
  }
  properties          => {
    myproperty => 'myvalue'
  },
  local_repo          => '/home/maven/.m2/repository'
}

We can override the central repository with mirrors, whichb add repositories to the mirrors settings. The servers parameter configures each settings.xml server entry for user and password credentials.

With default_repo_config, we can add a repository that will be enabled for all Maven executions, including the aven puppet type. That would be necessary in order to check a remote repository for snapshots, as there is no snapshot repository defined by default in Maven.

The properties parameter is a hash with keys and values for the properties section of the settings, while local_repo overrides Maven default local repository location.

Another Maven file that can be configured to alter the Maven environment variables is $HOME/.mavenrc with the maven::environment class. The .mavenrc is sourced by the Apache Maven script for each run.

maven::environment { 'env-maven-user' :
  user                 => 'maven',
  maven_opts           => '-XX:MaxPermSize=256m',
  maven_path_additions => '/usr/local/bin'
}

Probably the module’s most useful functionality is the ability to download artifacts from Maven repositories. This requires having Maven correctly installed and configured, which can be done with the previous classes and definitions, and uses the Maven dependency:get plugin behind the scenes. The title of the maven resource is used as the file destination, and the user to run maven as can be set with the user parameter.

maven { "/tmp/maven-core-2.2.1.jar":
  id    => "org.apache.maven:maven-core:2.2.1:jar",
  repos => ["central::default::http://repo.maven.apache.org/maven2","http://mirrors.ibiblio.org/pub/mirrors/maven2"],
  user  => "maven",
}

With the optional parameter repos, we can define what repositories to download the dependencies from if not using the default Maven central. The parameter is in the form expected by the Maven dependency plugin, that is id::[layout]::url or just url, separated by a comma.

Or, a little more verbose:

maven { "/tmp/maven-core-2.2.1-sources.jar":
  groupid    => "org.apache.maven",
  artifactid => "maven-core",
  version    => "2.2.1",
  classifier => "sources",
  packaging  => "jar",
  user       => "maven",
}

Example usage

With some simple declarations we can install Maven in a node, downloading the Apache Maven binaries from apache.org and uncompressing them under /usr/local, and then download any file from the central Maven repo. An example is maven-core-2.2.1.jar, which is located in the repository under org.apache.maven groupId and maven-core artifactId.

# Install Maven
class { "maven::maven": } ->

maven { "/tmp/maven-core-2.2.1.jar":
  id => "org.apache.maven:maven-core:2.2.1:jar",
}

The usage of the shorter form groupId:artifactId:version:packaging allows us to be more concise, but we could do the same using the groupid, artifactid, version, packaging parameters of the maven type. Note that we are using the chain arrow (->) to explicitly install Maven before using it to download the jar file.

You should have a /tmp/maven-core-2.2.1.jar file with contents matching those of http://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.jar.

Conclusion

If you use Apache Maven this module comes in handy for installing and configuring it on any machine in a consistent and repeatable way. This module also consumes the output artifacts from the development process in later stages of product delivery without extra steps or re-packaging.

Please let us know if you have any issues with the module. We are looking for new ways to improve the module, such as removing the need for wget to be installed. We look forward to your feedback!

Learn More: