Webinar: Scaling Jenkins with Docker and Kubernetes

Webinar: Scaling Jenkins with Docker and Kubernetes

Check the video at DevOps.com

Scaling Docker with Kubernetes V1: Kubernetes, Google Container Engine, Jenkins at Scale

Scaling Docker with Kubernetes V1

My latest article at InfoQ, covering Kubernetes v1, Google Container Engine, Jenkins at scale and the Jenkins Kubernetes plugin.

Kubernetes is an open source project to manage a cluster of Linux containers as a single system, managing and running Docker containers across multiple hosts, offering co-location of containers, service discovery and replication control. It was started by Google and now it is supported by Microsoft, RedHat, IBM and Docker amongst others.

Building Docker images with Puppet

Docker-logoEverybody should be building Docker images! but what if you don’t want to write all those shell scripts, which is basically what the Dockerfile is, a bunch of shell commands in RUN declarations; or if you are already using some Puppet modules to build VMs?

It is easy enough to build a new Docker image from Puppet manifests. For instance I have built this Jenkis slave Docker image, so here are the steps.

The Devops Israel team has built a number of Docker images on CentOS with Puppet preinstalled, so that is a good start.

FROM devopsil/puppet:3.5.1

Otherwise you can just install Puppet in any bare image using the normal installation instructions. Something to have into account is that Docker images are quite simple and may not have some needed packages installed. In this case the centos6 image didn’t have tar installed and some things failed to run. In some CentOS images the centosplus repo needs to be enabled for the installation to succeed.

FROM centos:centos6
RUN rpm --import https://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs && \
    rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

# Need to enable centosplus for the image libselinux issue
RUN yum install -y yum-utils
RUN yum-config-manager --enable centosplus

RUN yum install -y puppet tar

Once Puppet is installed we can apply any manifest to the server, we just need to put the right files in the right places. If we need extra modules we can copy them from the host, maybe using librarian-puppet to manage them. Note that I’m avoiding to run librarian or any tool in the image, as that would require installing extra packages that may not be needed at runtime.

ADD modules/ /etc/puppet/modules/

The main manifest can go anywhere but the default place is into /etc/puppet/manifests/site.pp. Hiera data default configuration goes into /var/lib/hiera/common.yaml.

ADD site.pp /etc/puppet/manifests/
ADD common.yaml /var/lib/hiera/common.yaml

Then we can just run puppet apply and check that no errors happened

RUN puppet apply /etc/puppet/manifests/site.pp --verbose --detailed-exitcodes || [ $? -eq 2 ]

After that it’s the usual Docker CMD configuration. In this case we call Jenkins slave jar from a shell script that handles some environment variables, with information about the Jenkins master, so it can be overriden at runtime with docker run -e.

ADD cmd.sh /cmd.sh

#ENV JENKINS_MASTER http://jenkins:8080

CMD su jenkins-slave -c '/bin/sh /cmd.sh'

The Puppet configuration is simple enough

node 'default' {
  package { 'wget':
    ensure => present
  } ->
  class { '::jenkins::slave': }

and Hiera customizations, using a patched Jenkins module for this to work.

# Jenkins slave
jenkins::slave::ensure: stopped
jenkins::slave::enable: false

And that’s all, you can see the full source code at GitHub. If you are into Docker check out this IBM research paper comparing virtual machines (KVM) and Linux containers (Docker) performance.

Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days

agiletdThis week I’m in Postdam/Berlin giving a talk Infrastructure testing with Jenkins, Puppet and Vagrant at Agile Testing Days. Showing examples of using Puppet, Vagrant and other tools to implement a source code to production continuous delivery cycle.

Slides are up in SlideShare, and source code is available at GitHub.

Extend Continuous Integration to automatically test your infrastructure.

Continuous Integration can be extended to test deployments and production environments, in a Continuous Delivery cycle, using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations, and test the infrastructure the same way continuous integration tools do with developers’ code.

Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services, … in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.

Using Vagrant, a command line automation layer for VirtualBox, we can easily spin off virtual machines with the same configuration as production servers, run our test suite, and tear them down afterwards.

We will show how to set up automated testing of an application and associated infrastructure and configurations, creating on demand virtual machines for testing, as part of your continuous integration process.

Hudson and Jenkins, can brands and trademarks affect an open source project?

Maybe you haven’t heard yet (where have you been!), but Hudson, the Continuous Integration server, is being renamed to Jenkins.

Sacha Labourey sheds a bit of light on the background that motivates the changes, but let’s just say that the community wants to move in a direction that Oracle is against, and Oracle claims the ownership of the Hudson trademark, so the community decides to move the source code to another hosting and use a different name for this fork.

So you have on one side Oracle, which AFAIK is doing few contributions to the project, and most of the people doing actual contributions on the other side.

Now, playing devil’s advocate, what if Oracle decides to continue Hudson, by, for instance, just merging all changes to Jenkins back into Hudson. No non-Oracle committers, a bit of development on their side if needed, just a close copy of Jenkins, but with the Hudson brand.

Would that succeed?

Of course you know that the community-endorsed project is now Jenkins. How many people out there would know that too?

Would current users in big corporations be aware of the status of the project?

Would it be possible for a giant like Oracle to use its marketing machinery to promote Hudson as an Oracle product and build some Hudson-branded products on top by just owning the trademark?

Could somebody pull a trick on the open source world as Eddie Murphy does in The Distinguished Gentleman? Vote for Jeff Johnson, the name you know!

I would hope that’s not the case, specially because Hudson doesn’t seem to play a big role in Oracle product map, but if it were otherways… I wouldn’t be so sure.