Kubernetes Plugin for Jenkins 0.12

Includes declarative pipeline support (note that you need Jenkins 2.66+ for it to work) and lots of bug fixes

The full changelog:

  • Add an experimental Declarative Agent extension for Kubernetes JENKINS-41758 #127
  • Implement Port mapping #165
  • Support idleMinutes field in pipeline #154
  • Add command liveness probe support #158
  • Add toggle for node usage mode #158
  • Add namespace support on PodTemplate.
  • Make PodTemplate optional within pipeline JENKINS-42315
  • Make Slave Jenkins connection timeout configurable #141
  • Fix durable pipeline PID NumberFormatException JENKINS-42048 #157
  • Don’t provision nodes if there are no PodTemplates set to usage mode Normal #171
  • Refactoring add/set methods in PodTemplate #173
  • Delete the build pod after we have finished with the template block #172
  • Default to use the kubernetes.default.svc.cluster.local endpoint
  • Do not print stack trace on ConnectException
  • Upgrade kubernetes client to 2.3.1 JENKINS-44189
  • Step namespace should have priority over anything else #161
  • Wait for pod to exist up to 60 seconds before erroring #155
  • Catch IOException on ContainerExecProc#kill
  • Do not print stack trace on connection exception
  • Restore random naming for pipeline managed pod templates.
  • Dir context is not honored by shell step JENKINS-40925 #146
  • Limit pod name to 63 characters, and change the randomly generated string #143
  • Fix workingDir inheritance error #136
  • Use name instead of label for the nesting stack #137
  • Exception in configure page when ‘Kubernetes URL’ isn’t filled JENKINS-45282 #174
  • kubectl temporary config file should work where Jenkins project contains spaces #178
  • Thread/connection leak #177

Running a JVM in a Container Without Getting Killed

No pun intended

The JDK 8u131 has backported a nice feature in JDK 9, which is the ability of the JVM to detect how much memory is available when running inside a Docker container.

I have talked multiple times about the problems running a JVM inside a container, how it will default in most cases to a max heap of 1/4 of the host memory, not the container.

For example in my machine

$ docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 444.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Wait, WAT? I set a container memory of 100MB and my JVM sets a max heap of 444M ? It is very likely that it is going to cause the Kernel to kill my JVM at some point.

Let’s try the JDK 8u131 with the experimental option -XX:+UseCGroupMemoryLimitForHeap

$ docker run -m 100MB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 44.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Ok this makes more sense, the JVM was able to detect the container has only 100MB and set the max heap to 44M.

Let’s try in a bigger container

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 228.00M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Mmm, now the container has 1GB but JVM is only using 228M as max heap. Can we optimize this even more, given that nothing else other than the JVM is running in the container? Yes we can!

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 910.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Using -XX:MaxRAMFraction we are telling the JVM to use available memory/MaxRAMFraction as max heap. Using -XX:MaxRAMFraction=1 we are using almost all the available memory as max heap.

Docker Registry with Let’s Encrypt Certificate

A one-liner to run a SSL Docker registry generating a Let’s Encrypt certificate.

This command will create a registry proxying the Docker hub, caching the images in a registry volume.

LetsEncrypt certificate will be auto generated and stored in the host dir as letsencrypt.json. You could also use a Docker volume to store it.

In order for the certificate generation to work the registry needs to be accessible from the internet in port 443. After the certificate is generated that’s no longer needed.

docker run -d -p 443:5000 --name registry \
  -v `pwd`:/etc/docker/registry/ \
  -v registry:/var/lib/registry \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
  -e REGISTRY_HTTP_HOST=https://docker.example.com \
  -e REGISTRY_HTTP_TLS_LETSENCRYPT_CACHEFILE=/etc/docker/registry/letsencrypt.json \
  -e REGISTRY_HTTP_TLS_LETSENCRYPT_EMAIL=admin@example.com \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
  registry:2

You can also create a config.yml in this dir and run the registry using the file instead of environment variables

version: 0.1
storage:
  filesystem:
http:
  addr: 0.0.0.0:5000
  host: https://docker.example.com
  tls:
    letsencrypt:
      cachefile: /etc/docker/registry/letsencrypt.json
      email: admin@example.com
proxy:
  remoteurl: https://registry-1.docker.io

Then run

docker run -d -p 443:5000 --name registry \
  -v `pwd`:/etc/docker/registry/ \
  -v registry:/var/lib/registry \
  registry:2

If you want to use this as a remote repository and not just for proxying, remove the proxy entry in the configuration

Running Docker containers as non root

Running containers as root is a bad practice, but many Docker images available in the Docker Hub have the user set to root by default, so what can we do about it?

TL;DR Use -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp for a typical Java image, plus any tool specific environment variable needed

Option 1 (build time):

Create a new derived image that creates a new user and changes the default to that one


FROM openjdk:8-jdk
RUN useradd --create-home -s /bin/bash user
WORKDIR /home/user
USER user

This is simple, but forces us to republish all these derived images, creating a maintenance nightmare.

Option 2 (runtime):

Use docker run -u option to choose what user to run the container as

docker run -ti --rm -u 1000 openjdk:8-jdk

This may work, but we can hit some issues, let’s see

$docker run -ti --rm -u 1000 openjdk:8-jdk git clone https://github.com/jenkinsci/docker
fatal: could not create work tree dir 'docker'.: Permission denied

Well, we obviously don’t have permissions to write to the default workdir, let’s fix it using -w and a dir that is writable, for instance /tmp

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.
fatal: unable to look up current user in the passwd file: no such user

 

Git does not like being run as an user that does not exist, so we need to pick one of the existing users

UPDATE: git 2.6.5+ removes this requirement, so we can run as any user even if it is not in the passwd file. For previous versions setting the GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL environment variables also works.

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false
messagebus:x:104:108::/var/run/dbus:/bin/false

The user nobody:65534 could be a good candidate, and as it is present in the default debian and alpine Docker images it will be present also in most images in the hub.

$ docker run -ti --rm -u 65534 -w /tmp openjdk:8-jdk \
git clone https://github.com/jenkinsci/docker
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.

Ok, that worked! Now let’s try to run something else, like a maven build


$ docker run -ti --rm -u 65534 -w /tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"
touch: cannot touch ‘/root/.m2/copy_reference_file.log’: Permission denied
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions?

This means entering in the domain of each tool and checking how to configure it. The maven docker image instructs us to use MAVEN_CONFIG and pass -Duser.home otherwise we would get an error [ERROR] Could not create local repository at /nonexistent/.m2/repository -> [Help 1]

Here is the working solution


$ docker run -ti --rm -u 65534 -w /tmp -e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn -Duser.home=/tmp package"

Can we generalize this a bit for other Java apps? yes! By using the env var _JAVA_OPTIONS we can pass the user.home property to any Java app.

$ docker run -ti --rm -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp \
-e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone https://github.com/jenkinsci/kubernetes-plugin.git && \
cd kubernetes-plugin && mvn package"

 

Jenkins Kubernetes Plugin 0.10 Released

The 0.10 release is mostly a bugfix release

Changelog for 0.10:

  • Fixing checkbox serialization by jelly views #110
  • Do not throw exceptions in the test configuration page #107
  • Upgrade to the latest kubernetes-client version. #106
  • feat: make pipeline support instanceCap field #102
  • Instantiating Kubernetes Client with proper config in Container Steps #104
  • Fix NPE when slaves are read from disk #103
  • [JENKINS-39867] Upgrade fabric8 to 1.4.26 #101
  • The pod watcher now checks readiness of the right pod. #97
  • Fix logic for waitUntilContainerIsReady #95
  • instanceCap is not used in pipeline #92
  • Allow nesting of templates for inheritance. #94
  • Wait until all containers are in ready state before starting the slave #93
  • Adding basic retention for idle slaves, using the idleTimeout setting properly #91
  • Improve the inheritFrom functionality to better cover containers and volumes. #84
  • Fix null pointer exceptions. #89
  • fix PvcVolume jelly templates path #90
  • Added tool installations to the pod template. #85
  • fix configmap volume name #87
  • set the serviceAccount when creating new pods #86
  • Read and connect timeout are now correctly used to configure the client. #82
  • Fix nodeSelector in podTemplate #83
  • Use the client’s namespace when deleting a pod (fixes a regression preventing pods to delete). #81

Next Week: DevOps Pro, Moscow and ApacheCON Europe, Seville

Next week is a busy one, will be speaking at DevOps Pro in Moscow then ApacheCON Europe in Seville. If you are around give me a shout!

And if you are organizing an event next year and are interested in me giving a talk, let me know.

From Monolith to Docker Distributed Applications

Docker is revolutionizing the way people think about applications and deployments. It provides a simple way to run and distribute Linux containers for a variety of use cases, from lightweight virtual machines to complex distributed micro-services architectures.

Containers allow to run services in isolation with a minimum performance penalty, increased speed, easier configuration and less complexity, making it ideal for continuous integration and continuous delivery based workloads. But migrating an existing application to a distributed micro-services architecture is no easy task, requiring a shift in the software development, networking and storage to accommodate the new architecture.

We will provide insight on our experience creating a Jenkins platform based on distributed Docker containers running on Apache Mesos and comparing other solutions as Kubernetes or Docker Swarm, applicable for all types of applications, but specially Java and JVM based nones.

CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos

As Continuous Integration and Continuous Delivery usage grows, the underlying infrastructure needs to grow too, scaling horizontally across multiple hosts and dynamically provisioned as needed.

Docker is revolutionizing the way people think about applications and deployments. Apache Mesos enables fault-tolerant distributed systems and allows managing a cluster of Linux servers as a single system, running Docker containers across multiple hosts, offering elasticity and machine abstractions.

By using Apache Mesos and Marathon, an Apache Mesos framework for long-running applications, the Jenkins Continuous Integration environment can be run at scale, dynamically.

Jenkins masters can be run on Docker containers distributed on Marathon, while the Jenkins Mesos and Docker plugins allow slaves and jobs to run in containers distributed across the multiple hosts, and isolating job execution.

 

Jenkins Kubernetes Plugin 0.9 Released

New features released in 0.9 include pipeline support and multiple containers per pod.

So now it is possible to define all the containers used in a job in your Jenkinsfile, for instance building a Maven project and a golang project in the same node without having to create any specific Docker image \o/

podTemplate(label: 'mypod', containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'golang', image: 'golang:1.6.3-alpine', ttyEnabled: true, command: 'cat')
  ],
  volumes: [secretVolume(secretName: 'shared-secrets', mountPath: '/etc/shared-secrets')]) {

  node ('mypod') {
    stage 'Get a Maven project'
    git 'https://github.com/jenkinsci/kubernetes-plugin.git'
    container('maven') {
      stage 'Build a Maven project'
      sh 'mvn clean install'
    }

    stage 'Get a Golang project'
    git url: 'https://github.com/hashicorp/terraform.git'
    container('golang') {
      stage 'Build a Go project'
      sh """
      mkdir -p /go/src/github.com/hashicorp
      ln -s `pwd` /go/src/github.com/hashicorp/terraform
      cd /go/src/github.com/hashicorp/terraform && make core-dev
      """
    }

  }
}

Changelog:

  • Make it possible to define more than one container inside a pod.
  • Add new pod template step which allows defining / overriding a pod template from a pipeline script.
  • Introduce pipeline step that allows choosing one of the containers of the pod and have all ‘sh’ steps executed there.
  • allow setting dynamic pod volumes in pipelines
  • Add support for persistent volume claims
  • Add support for containerEnvVar’s in pipelines
  • [JENKINS-37087] Handle multiple labels per pod correctly
  • [JENKINS-37087] Iterate over all matching templates
  • Fix slave description and labels
  • [JENKINS-38829] Add help text for Kubernetes server certificate
  • #59: Allow blank namespace and reuse whatever is discovered by the client.
  • Ensure instanceCap defaults to unlimited
  • Add Jenkins computer name to the container env vars
  • Split arguments having quotes into account
  • Allow the user to enable pseudo-TTY on container level.
  • Use provided arguments without forcing jnlpmac and name into them. Provide placeholders for jnlpmac and name for the user to use. Fallback container uses as default arguments jnlpmac and name.
  • Split volume classes into their own package (#77)