Security Testing Using Infrastructure-As-Code

Agile RecordArticle originally published at Agile Record magazine Issue #17 Security Testing in an Agile Environment. Can be downloaded for free as a PDF.

Security Testing Using Infrastructure-As-Code

Infrastructure-As-Code means that infrastructure should be treated as code – a really powerful concept. Server configuration, packages installed, relationships with other servers, etc. should be modeled with code to be automated and have a predictable outcome, removing manual steps prone to errors. That doesn’t sound bad, does it?

The goal is to automate all the infrastructure tasks programmatically. In an ideal world you should be able to start new servers, configure them, and, more importantly, be able to repeat it over and over again, in a reproducible way, automatically, by using tools and APIs.

Have you ever had to upgrade a server without knowing whether the upgrade was going to succeed or not for your application? Are the security updates going to affect your application? There are so many system factors that can indirectly cause a failure in your application, such as different kernel versions, distributions, or packages.

When you have a decent set of integration tests it is not that hard to make changes to your infrastructure with that safety net. There are a number of tools designed to make your life easier, so there is no need to tinker with bash scripts or manual steps prone to error.

We can find three groups of tools:

  • Provisioning tools, like Puppet or Chef, manage the configuration of servers with packages, services, config files, etc. in a reproducible way and over hundreds of machines.
  • Virtual Machine automation tools, like Vagrant, enable new virtual machines to be started easily in different environments, from virtual machines in VirtualBox or VMware to cloud providers such as Amazon AWS or Rackspace, and then provision them with Puppet or Chef.
  • Testing tools, like rspec, Cucumber, or Selenium, enable unit and integration tests to be written that verify that the server is in a good state continuously as part of your continuous integration process.

Vagrant

Learning Puppet can be a tedious task, such as getting up the different pieces (master, agents), writing your first manifests, etc. A good way to start is to use Vagrant, which started as an Oracle VirtualBox command line automation tool, and allows you to create new VMs locally or on cloud providers and provision them with Puppet and Chef easily.

Vagrant projects are composed of base boxes, specifically configured for Vagrant with Puppet/Chef, vagrant username and password, and any customizations you may want to add, plus the configuration to apply to those base boxes defined with Puppet or Chef. That way we can have several projects sharing the same base boxes where the Puppet/Chef definitions are different. For instance, a database VM and a web server VM can both use the same base box, i.e. a CentOS 6 minimal server, and just have different Puppet manifests. When Vagrant starts them up it will apply the specific configuration. That also allows you to share boxes and configuration files across teams. For instance, one base box with the Linux flavor can be used in a team, and in source control we can have just the Puppet manifests to apply for the different configurations that anybody from Operations to Developers can use. If a problem arises in production, a developer can quickly instantiate a equivalent environment using the Vagrant and Puppet configuration, making a different environment’s issues easy to reproduce.

There is a list of available VMs or base boxes ready to use with Vagrant at www.vagrantbox.es, but you can build your own and share it anywhere. For VirtualBox they are just (big) VM files that can be easily built using VeeWee (https://github.com/jedi4ever/veewee) or by changing a base box and rebundling it with Packer (http://www.packer.io).

Usage

Once you have installed Vagrant (http://docs.vagrantup.com/v2/installation/index.html) and VirtualBox (https://www.virtualbox.org/) you can create a new project.

Vagrant init will create a sample Vagrantfile, the project definition file that can be customized.

$ vagrant init myproject

Then in the Vagrantfile you can change the default box settings and add basic Puppet provisioning.

config.vm.box = "CentOS-6.4-x86_64-minimal"
config.vm.box_url = "https://repo.maestrodev.com/archiva/repository/public-releases/com/maestrodev/vagrant/CentOS/6.4/CentOS-6.4-x86_64-minimal.box"

# create a virtual network so we can access the vm by ip
config.vm.network "private_network", ip: "192.168.33.13"
config.vm.hostname = "qa.acme.local"
config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "manifests"
  puppet.manifest_file = "site.pp"
  puppet.module_path = "modules"
  end

In manifests/site.pp you can try any puppet code, i.e. create a file

node 'qa.acme.local' {
  file { '/root/secret':
  mode => '0600',
  owner => 'root',
  content => 'secret file, for root eyes only',
  }
}

Vagrant up will download the box the first time, start the VM, and apply the configuration defined in Puppet.

$ vagrant up

vagrant ssh will open a shell into the box. Under the hood, vagrant is redirecting a host port to vagrant box 22.

$ vagrant ssh

If you make any changes to the Puppet manifests you can rerun the provisioning step.

$ vagrant provision

The vm can be suspended and resumed at any time

$ vagrant suspend
$ vagrant resume

and later on destroyed, which will delete all the VM files.

$ vagrant destroy

And then we can start again from scratch with vagrant up getting a completely new vm where we can make any mistakes!

Puppet

In Puppet we can configure any aspect of a server: packages, files, permissions, services, etc. You have seen how to create a file, now let’s see an example of configuring Apache httpd server and the Linux iptables firewall to open a port.

First we need the Puppet modules to manage httpd and the firewall rules to avoid writing all the bits and pieces ourselves. Modules are Puppet reusable components that you can find at the Puppet Forge (http://forge.puppetlabs.com/) or typically in GitHub. To install these two modules into the vm, run the following commands that will download the modules and install them in the /etc/puppet/modules directory.

vagrant ssh -c "sudo puppet module install --version 0.9.0 puppetlabs/apache"
vagrant ssh -c "sudo puppet module install --version 0.4.2 puppetlabs/firewall"

You can find more information about the Apache (http://forge.puppetlabs.com/puppetlabs/apache/0.9.0) and the Firewall (http://forge.puppetlabs.com/puppetlabs/firewall/0.4.2) modules in their Forge pages. We are just going to add some simple examples to the manifests/site.pp to install the Apache server with a virtual host that will listen in port 80.

node 'qa.acme.local' {

  class { 'apache': }

  # create a virtualhost

  apache::vhost { "${::hostname}.local":
    port => 80,
    docroot => '/var/www',
  }
  }

Now if you try to access this server in port 80 you will not be able to, as iptables is configured by default to block all incoming connections. Try accessing http://192.168.33.13 (the ip we configured previously in the Vagrantfile for the private virtual network) and see for yourself.

To open the firewall, we need to open the port explicitly in the manifests/site.pp by adding

firewall { '100 allow apache':
  proto => 'tcp',
  port => '80',
  action => 'accept',
  }

and running vagrant provision again. Now you should see Apache’s default page in http://192.168.33.13.

So far we have created a virtual machine where the apache server is automatically installed and the firewall open. You could start from scratch at any time by running vagrant destroy and vagrant up again.

Testing

Let’s write some tests to ensure that everything is working as expected. We are going to use Ruby as the language of choice.

Unit testing with rspec-puppet

rspec-puppet (http://rspec-puppet.com/) is a rspec extension that allows to easily unit test Puppet manifests.

Create a spec/spec_helper.rb file to add some shared config for all the specs

require 'rspec-puppet'

RSpec.configure do |c|
  c.module_path = 'modules'
  c.manifest_dir = 'manifests'
  end

and we can start creating unit tests for the host that we defined in Puppet.

# spec/hosts/qa_spec.rb

require 'spec_helper'

describe 'qa.acme.local' do

  # test that the httpd package is installed

  it { should contain_package('httpd') }

  # test that there is a firewall rule set to 'accept'

  it { should contain_firewall('100 allow apache').with_action('accept') }

  # ensure that there is only one firewall definition

  it { should have_firewall_resource_count(1) }

  end

After installing rspec-puppet gem install rspec-puppet, you can run rspec to execute the tests.

...

Finished in 1.4 seconds

3 examples, 0 failures

Success!

Integration testing with Cucumber

Unit testing is fast and can catch a lot of errors quickly, but how can we check that the machine is actually configured as we expected?

Let’s use Cucumber (http://cukes.info/), a BDD tool, to create an integration test that checks whether a specific port is open in the virtual machine we started.

Create a features/smoke_tests.feature file with:

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port
Then the "apache" service should be listening on port "80"

Install Cucumber gem install cucumber and run cucumber. The first run will output a message saying that the step definition has not been created yet.

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port # features/smoke_tests.feature:4
Then the "apache" service should be listening on port "80" # features/smoke_tests.feature:5

1 scenario (1 undefined)

1 step (1 undefined)

0m0.001s

You can implement step definitions for undefined steps with these snippets:

Then(/^the "(.*?)" service should be listening on port "(.*?)"$/) do |arg1, arg2|
  pending # express the regexp above with the code you wish you had
  end

So let’s create a features/step_definitions/tcp_ip_steps.rb file that implements our service should be listening on port step by opening a TCP socket.

Then /^the "(.*?)" service should be listening on port "(.*?)"$/ do |service, port|
  host = URI.parse(ENV['URL']).host
  begin
    s = TCPSocket.new(host, port)
    s.close
    rescue Exception => error
    raise("#{service} is not listening at #{host} on port #{port}")
  end
  end

And rerun Cucumber, this time using an environment variable URL to specify where the machine is running, as used in the step definition URL=http://192.168.33.13 cucumber.

Feature: Smoke tests
Smoke testing scenarios to make sure all system components are up and running.

Scenario: Services should be up and listening to their assigned port # features/smoke_tests.feature:4
Then the "apache" service should be listening on port "80" # features/step_definitions/tcp_ip_steps.rb:1

1 scenario (1 passed)

1 step (1 passed)

0m0.003s

Success! The port is actually open in the virtual machine.

Wash, rinse, repeat

This was a small example of what can be achieved using Infrastructure-As-Code and automation tools such as Puppet and Vagrant combined with standard testing tools like rspec or Cucumber. When a continuous integration tool like Jenkins is thrown into the mix to run these tests continuously, the result is an automatic end-to-end solution that tests systems as any other code, avoiding regressions and enabling Continuous Delivery (http://blog.csanchez.org/2013/11/12/continuous-delivery-with-maven-puppet-and-tomcat-video-from-apachecon-na-2013/) – automation all the way from source to production.

A more detailed example can be found in my continuous-delivery project at GitHub (https://github.com/carlossg/continuous-delivery).

Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days

agiletdThis week I’m in Postdam/Berlin giving a talk Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days. Showing examples of using Puppet, Vagrant and other tools to implement a source code to production continuous delivery cycle.

Slides are up in SlideShare, and source code is available at GitHub.

Extend Continuous Integration to automatically test your infrastructure.

Continuous Integration can be extended to test deployments and production environments, in a Continuous Delivery cycle, using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations, and test the infrastructure the same way continuous integration tools do with developers’ code.

Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services, … in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Using Vagrant, a command line automation layer for VirtualBox, we can easily spin off virtual machines with the same configuration as production servers, run our test suite, and tear them down afterwards.

We will show how to set up automated testing of an application and associated infrastructure and configurations, creating on demand virtual machines for testing, as part of your continuous integration process.

Testing puppet modules

Puppet Labs logoThere are several steps depending on how much involved the tests are, what parts are tested and, of course, how long it takes to run the tests.

For unit testing we use rspec puppet, and we can check that our manifests and modules compile and contain the expected values. It can be used to test that specific types, classes or definitions are in the compiled catalog and that the parameters math the expectations.

Later on we can do some integration testing starting a new VM with Vagrant and checking that there are no errors in the provisioning, as well as checking that some conditions are met.

For rspec-puppet, PuppetLabs has created a project called puppetlabs_spec_helper that let’s us avoid writing a bunch of boilerplate. A missing point though is that it only allows to use modules for testing from git. If you’re already using librarian-puppet (and you should!) you can easily use the same Puppetfile for deploying modules and to test them. Doing otherwise sounds like a bit of useless testing, you could end with different versions in different development machines, CI server, puppet master,… So just add a call to librarian puppet in your rakefile to populate the rspec-puppet fixtures before running the specs.

Unfortunately rspec-puppet doesn’t work with Puppet 3.0.x and  at least Puppet 3.1.0-rc1 is required. It was a bit of a setback when we moved to Puppet 3 and started using hiera, which is proving to be very useful to have simpler manifests and external data injected for our Maestro installations with Puppet from scratch.

You can also use the same Puppetfile to start Vagrant boxes with the exact same version of the modules. We are using Cucumber and Aruba to execute vagrant, provision the VM with puppet and check several things, like open ports, services up,… but that’s a different story :-)

Example

In this puppet-for-java-devs project you will find the bits that showcase all these tools integrated. It includes definition of a 3-tier system with Puppet definitions for a postgresql database, tomcat nodes with a war installed and apache nodes fronting them.

Install all required gems

bundle install

Install all Puppet modules with Puppet Librarian

librarian-puppet install

Run the specs with puppet-rspec

bundle exec rake

Start all the vms with Vagrant

vagrant up

Rakefile

require 'bundler'
Bundler.require(:rake)
require 'rake/clean'

require 'puppetlabs_spec_helper/rake_tasks'

CLEAN.include('modules', 'spec/fixtures/', 'doc')
CLOBBER.include('.tmp', '.librarian')

task :librarian_spec_prep do
 sh "librarian-puppet install"
end
task :spec_prep => :librarian_spec_prep

task :default => [:spec]

Puppetfile for librarian-puppet

forge 'http://forge.puppetlabs.com'

mod 'puppetlabs/java', '0.1.6'
mod 'puppetlabs/apache', '0.4.0'
mod 'inkling/postgresql', '0.2.0'
mod 'puppetlabs/firewall', '0.0.4'
mod 'tomcat', :git => 'https://github.com/carlossg/puppet-tomcat.git', :ref => 'centos'
mod 'maestrodev/maven', '1.x'
mod 'stahnma/epel', '0.0.2'
mod 'maestrodev/avahi', '1.x'
mod 'other', :path => 'mymodules/other'

tomcat_spec.rb with rspec-puppet

require 'spec_helper'

describe 'tomcat1.acme.com' do
  let(:facts) { {:osfamily => 'RedHat', :operatingsystem => 'CentOS', :operatingsystemrelease => 6.3} }

  it { should contain_class('java').with_distribution /openjdk/ }

  it "configure webapp" do
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT.war')
    should contain_maven('/srv/tomcat/appfuse/webapps/ROOT/WEB-INF/lib/postgresql-9.1-901.jdbc4.jar')
  end
end

Vagrantfile

Vagrant::Config.run do |config|
  config.vm.box = "CentOS-6.3-x86_64-minimal"
  config.vm.box_url = "https://dl.dropbox.com/u/7225008/Vagrant/CentOS-6.3-x86_64-minimal.box"

  config.vm.customize ["modifyvm", :id, "--rtcuseutc", "on"] # use UTC clock https://github.com/mitchellh/vagrant/issues/912

  # db server
  config.vm.define :db do |config|
    config.vm.host_name = "db.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "db"] # name for VirtualBox GUI
    config.vm.forward_port 5432, 5432
    config.vm.network :hostonly, "192.168.33.10"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # tomcat server
  config.vm.define :tomcat1 do |config|
    config.vm.host_name = "tomcat1.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "tomcat1"] # name for VirtualBox GUI
    config.vm.forward_port 8080, 8081
    config.vm.network :hostonly, "192.168.33.11"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

  # web server
  config.vm.define :www do |config|
    config.vm.host_name = "www.acme.local"
    config.vm.customize ["modifyvm", :id, "--name", "www"] # name for VirtualBox GUI
    config.vm.forward_port 80, 8080
    config.vm.network :hostonly, "192.168.33.12"
    config.vm.provision :puppet do |puppet|
      puppet.module_path = "modules"
      puppet.manifest_file = "site.pp"
    end
  end

end

Automatically download and install VirtualBox guest additions in Vagrant

So, are you already using Vagrant to manage your VirtualBox VMs?

Then you probably have realized already how annoying is to keep the VBox guest additions up to date in your VMs.

Don’t worry, you can update them with just one command or automatically on each start using the Vagrant-vbguest plugin.

Installation

Requires vagrant 0.9.4 or later (including 1.0)

Since vagrant v1.0.0 the prefered installation method for vagrant is using the provided packages or installers.

Therefore if you installed Vagrant as a package (rpm, deb, dmg,…)

vagrant gem install vagrant-vbguest

Or if you installed vagrant using RubyGems (gem install vagrant):

gem install vagrant-vbguest

Usage

By default the plugin will check what version of the guest additions is installed in the VM every time it is started with vagrant start. Note that it won’t be checked when resuming a box.

In any case, it can be disabled in the Vagrantfile

Vagrant::Config.run do |config|
  # set auto_update to false, if do NOT want to check the correct additions
  # version when booting this machine
  config.vbguest.auto_update = false
end

If it detects an outdated version, it will automatically install the matching version from the VirtualBox installation, located at

  • linux : /usr/share/virtualbox/VBoxGuestAdditions.iso
  • Mac : /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso
  • Windows : %PROGRAMFILES%/Oracle/VirtualBox/VBoxGuestAdditions.iso

The location can be overridden with the iso_path parameter in your Vagrantfile, and can point to a http server

Vagrant::Config.run do |config|
  config.vbguest.iso_path = "#{ENV['HOME']}/Downloads/VBoxGuestAdditions.iso"
  # or
  config.vbguest.iso_path = "http://company.server/VirtualBox/$VBOX_VERSION/VBoxGuestAdditions.iso"
end

If you have disabled the automatic update, it still easy to manually update the VirtualBox Guest Additions version, just running from the command line

vagrant vbguest

Learning Puppet or Chef? Check out Vagrant!

If you are starting to use Puppet or Chef, you must have Vagrant.

Learning Puppet can be a tedious task, getting up the master, agents, writing your first manifests,… A good way to start is using Vagrant, an Oracle VirtualBox command line automation tool, that allows easy Puppet and Chef provisioning on VirtualBox VMs.

Vagrant projects are composed by base boxes, specifically configured for Vagrant with Puppet/Chef, vagrant username and password, and anything else you may want to add, plus the configuration to apply to those base boxes, defined with Puppet or Chef. That way we can have several projects using the same base boxes shared where the only difference are the Puppet/Chef definitions. For instance a database VM and a web server VM can both use the same base box and just have different Puppet manifests, and when Vagrant starts them, it will apply the specific configuration. That also allows to share boxes and configuration files across teams, for instance having one base box with the Linux flavor used in a team, we can just have in source control the Puppet manifests to apply for the different configurations that anybody from Operations to Developers can use.

There is a list of available VMs or base boxes ready to use with Vagrant at www.vagrantbox.es. But you can build your own and share it anywhere, as they are just (big) VirtualBox VM files, easily using VeeWee, or changing a base box and rebundling it with vagrant package.

Usage

Once you have installed Vagrant and VirtualBox.

Vagrant init will create a sample Vagrantfile, the project definition file that can be customized.

$ vagrant init myproject

Then in the Vagrantfile you can change the default box settings, and add basic Puppet provisioning

config.vm.box = "centos-6"
config.vm.box_url = "https://vagrant-centos-6.s3.amazonaws.com/centos-6.box"

config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "manifests"
  puppet.manifest_file = "site.pp"
end

In manifests/site.pp you can try any puppet manifest.

file { '/etc/motd':
  content => 'Welcome to your Vagrant-built virtual machine! Managed by Puppet.\n'
}

Vagrant up will download the box the first time, start the VM and apply any configuration defined in Puppet

$ vagrant up

vagrant ssh will open a shell into the box. Under the hood vagrant is redirecting the host port 2222 to the vagrant box 22

$ vagrant ssh

The vm can be suspended and resumed at any time

$ vagrant suspend
$ vagrant resume

and later on destroyed, which will delete all the VM files.

$ vagrant destroy

And then we can start again from scratch with vagrant up getting a completely new vm where we can make any mistakes :)

From Dev to DevOps slides from Agile Spain

Conferencia Agile Spain 2011Updated slides from my “From Dev to DevOps” presentation at the Agile Spain conference in Castellon on October 20th. Thanks to all the attendees for the questions and feedback!

Just some updates on Vagrant, VeeWee, Geppetto, and Puppet-Maven. Next stop, ApacheCON Vancouver!

UPDATE The video is also available (in Spanish) at the UJI web server as Flash and WMV.

From Dev to DevOps slides from Apache Barcamp Spain

Here are the slides from my “From Dev to DevOps” presentation at the Apache Barcamp Spain in Seville on October 8th. Not all that useful without the talking and hand waving :)

I’ll be presenting it too at the Agile Spain conference on Thursday, with new slides, and adding some more info on Vagrant, VeeWee, Geppetto, and Puppet-Maven, just ten days after, things evolve really fast! Then, on to present at ApacheCON in Vancouver.

I’ll hopefully find the time to publish here at some point, in the meantime, there’s a good summary about the tools, Setup your devops playground with Puppet, Vagrant & co by Arnaud Heritier.