Running Docker containers as non root

Running containers as root is a bad practice, but many Docker images available in the Docker Hub have the user set to root by default, so what can we do about it?

TL;DR Use -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp for a typical Java image, plus any tool specific environment variable needed

Option 1 (build time):

Create a new derived image that creates a new user and changes the default to that one


FROM openjdk:8-jdk
RUN useradd --create-home -s /bin/bash user
WORKDIR /home/user
USER user

This is simple, but forces us to republish all these derived images, creating a maintenance nightmare.

Option 2 (runtime):

Use docker run -u option to choose what user to run the container as

docker run -ti --rm -u 1000 openjdk:8-jdk

This may work, but we can hit some issues, let’s see

$docker run -ti --rm -u 1000 openjdk:8-jdk git clone https://github.com/jenkinsci/docker
fatal: could not create work tree dir 'docker'.: Permission denied

Well, we obviously don’t have permissions to write to the default workdir, let’s fix it using -w and a dir that is writable, for instance /tmp

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.
fatal: unable to look up current user in the passwd file: no such user

 

Git does not like being run as an user that does not exist, so we need to pick one of the existing users

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false
messagebus:x:104:108::/var/run/dbus:/bin/false

The user nobody:65534 could be a good candidate, and as it is present in the default debian and alpine Docker images it will be present also in most images in the hub.

$ docker run -ti --rm -u 65534 -w /tmp openjdk:8-jdk \
git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.

Ok, that worked! Now let’s try to run something else, like a maven build


$ docker run -ti --rm -u 65534 -w /tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"
touch: cannot touch ‘/root/.m2/copy_reference_file.log’: Permission denied
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions?

This means entering in the domain of each tool and checking how to configure it. The maven docker image instructs us to use MAVEN_CONFIG and pass -Duser.home otherwise we would get an error [ERROR] Could not create local repository at /nonexistent/.m2/repository -> [Help 1]

Here is the working solution


$ docker run -ti --rm -u 65534 -w /tmp -e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn -Duser.home=/tmp package"

Can we generalize this a bit for other Java apps? yes! By using the env var _JAVA_OPTIONS we can pass the user.home property to any Java app.

$ docker run -ti --rm -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp \
-e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"

 

JavaOne: From Monolith to Docker Distributed Applications

I’ll be speaking again this year at JavaOne: From Monolith to Docker Distributed Applications, sharing our experience running the Jenkins platform on Docker containers using Apache Mesos.

You can also find me in the CloudBees booth in the exhibitors area.

Docker is revolutionizing the way people think about applications and deployments. It provides a simple way to run and distribute Linux containers for a variety of use cases, from lightweight virtual machines to complex distributed microservice architectures.

But migrating an existing Java application to a distributed microservice architecture is no easy task, requiring a shift in the software development, networking, and storage to accommodate the new architecture.

This presentation provides insights into the experience of the speaker and his colleagues in creating a Jenkins platform based on distributed Docker containers running on Apache Mesos and Marathon and applicable to all types of applications, especially Java- and JVM-based ones.

Next Events: DevOpsPro Vilnius, MesosCon, Boulder JAM & Docker meetups, Open DevOps Milan

I’ll be traveling in the following weeks, speaking at

DevOpsPro in Vilnius, Lithuania: From Monolith to Docker Distributed Applications (May 26th)

MesosCon North America in Denver, CO: CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos (June 1st)

Jenkins Area Meetup and Docker Boulder meetup in Boulder, CO: CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos (June 2nd)

Open DevOps in Milan, Italy: Continuous Delivery and the DevOps Way (June 22nd)

If you are around just ping me!

Downloading artifacts from a Maven repository with Ansible

AnsibleLogoAn example of downloading artifacts from a Maven repository using Ansible, including a prebuilt Docker image.

Prerequisites

Install JDK and Maven using existing Ansible modules

ansible-galaxy install geerlingguy.java
ansible-galaxy install https://github.com/silpion/ansible-maven.git
- hosts: localhost

 roles:
 - { role: ansible-maven }
 - { role: geerlingguy.java }

 vars:
 java_packages:
 - java-1.7.0-openjdk

Example

From mvn.yml, download any number of Maven artifacts optionally from different repositories

- hosts: localhost

 vars:
 mvn_artifacts:
 - id: org.apache.maven:maven-core:2.2.1:jar:sources
 dest: /tmp/test.jar
 # params: -U # update snapshots
 # repos:
 # - http://repo1.maven.apache.org/maven2
 
 tasks:
 - name: copy maven artifacts
 command: mvn {{ item.params | default('') }} org.apache.maven.plugins:maven-dependency-plugin:get -Dartifact={{ item.id }} -Ddest={{ item.dest }} -Dtransitive=false -Pansible-maven -DremoteRepositories={{ item.repos | default(['http://repo1.maven.apache.org/maven2']) | join(",") }}
 with_items: mvn_artifacts

Docker

An image with Ansible, JDK and Maven preinstalled is available at csanchez/ansible-maven.

Scaling Docker with Kubernetes

kubernetesI have published a new article on InfoQ, Scaling Docker with Kubernetes, where I describe the Kubernetes project and how it allows to run Docker containers across multiple hosts.

Kubernetes is an open source project to manage a cluster of Linux containers as a single system, managing and running Docker containers across multiple hosts, offering co-location of containers, service discovery and replication control.

Included in the article there is an example of scaling Jenkins in a master – multiple slaves architecture, where the slaves are running in Kubernetes. When I finally find the time I will implement a Jenkins Kubernetes plugin that would handle the scaling automatically.

Continuous Discussion panel about Agile, Continuous Delivery, DevOps

electriccloudLast week I participated as a panelist in the Continuous Discussions talk hosted by Electric Cloud, and the recording is now available. A bit long but there are some good points in there.

Some excerpts from twitter

@csanchez: “How fast can your tests absorb your debs agility” < and your Ops, and your Infra?

@cobiacomm: @orfjackal says ‘hard to do agile when the customer plan is to release once per year’

@sunandaj17: It’s not just about the tools: is a matter of team policies & conventions) & it relies on more than 1 kind of tool

@eriksencosta: “You can’t outsource Agile”.

@cobiacomm: biggest agile obstacles -> long regression testing cycles, unclear dependencies, and rebuilding the wheel

The panelists:

Andrew Rivers – blog.andrewrivers.co.uk
Carlos Sanchez – @csanchez   |  http://csanchez.org
Chris Haddad – @cobiacomm
Dave Josephsen – @djosephsen
Eriksen Costa – @eriksencosta  |  blog.eriksen.com.br
Esko Luontola – @orfjackal  |  www.orfjackal.net
John Ryding – @strife25  |  blog.johnryding.com
Norm MacLennan – @nromdotcom  |  blog.normmaclennan.com
J. Randall Hunt – @jrhunt  |  blog.ranman.org
Sriram Narayan – @sriramnarayan  |  www.sriramnarayan.com
Sunanda Jayanth – @sunandaj17  |  http://blog.qruizelabs.com/

Hosts: Sam Fell (@samueldfell) and Anders Wallgren (@anders_wallgren) from Electric Cloud.

http://electric-cloud.com/blog/2014/10/c9d9-continuous-discussions-episode-1-recap/

Building Docker images with Puppet

Docker-logoEverybody should be building Docker images! but what if you don’t want to write all those shell scripts, which is basically what the Dockerfile is, a bunch of shell commands in RUN declarations; or if you are already using some Puppet modules to build VMs?

It is easy enough to build a new Docker image from Puppet manifests. For instance I have built this Jenkis slave Docker image, so here are the steps.

The Devops Israel team has built a number of Docker images on CentOS with Puppet preinstalled, so that is a good start.


FROM devopsil/puppet:3.5.1

Otherwise you can just install Puppet in any bare image using the normal installation instructions. Something to have into account is that Docker images are quite simple and may not have some needed packages installed. In this case the centos6 image didn’t have tar installed and some things failed to run. In some CentOS images the centosplus repo needs to be enabled for the installation to succeed.


FROM centos:centos6
RUN rpm --import https://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs && \
    rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

# Need to enable centosplus for the image libselinux issue
RUN yum install -y yum-utils
RUN yum-config-manager --enable centosplus

RUN yum install -y puppet tar

Once Puppet is installed we can apply any manifest to the server, we just need to put the right files in the right places. If we need extra modules we can copy them from the host, maybe using librarian-puppet to manage them. Note that I’m avoiding to run librarian or any tool in the image, as that would require installing extra packages that may not be needed at runtime.


ADD modules/ /etc/puppet/modules/

The main manifest can go anywhere but the default place is into /etc/puppet/manifests/site.pp. Hiera data default configuration goes into /var/lib/hiera/common.yaml.


ADD site.pp /etc/puppet/manifests/
ADD common.yaml /var/lib/hiera/common.yaml

Then we can just run puppet apply and check that no errors happened


RUN puppet apply /etc/puppet/manifests/site.pp --verbose --detailed-exitcodes || [ $? -eq 2 ]

After that it’s the usual Docker CMD configuration. In this case we call Jenkins slave jar from a shell script that handles some environment variables, with information about the Jenkins master, so it can be overriden at runtime with docker run -e.


ADD cmd.sh /cmd.sh

#ENV JENKINS_USERNAME jenkins
#ENV JENKINS_PASSWORD jenkins
#ENV JENKINS_MASTER http://jenkins:8080

CMD su jenkins-slave -c '/bin/sh /cmd.sh'

The Puppet configuration is simple enough

node 'default' {
  package { 'wget':
    ensure => present
  } ->
  class { '::jenkins::slave': }
}

and Hiera customizations, using a patched Jenkins module for this to work.


# Jenkins slave
jenkins::slave::ensure: stopped
jenkins::slave::enable: false

And that’s all, you can see the full source code at GitHub. If you are into Docker check out this IBM research paper comparing virtual machines (KVM) and Linux containers (Docker) performance.