Serverless CI/CD with AWS ECS Fargate

Amazon AWS has recently launched ECS Fargate to “run containers without having to manage servers or clusters”.

So this got me interested enough to patch the Jenkins ECS plugin to run Jenkins agents as containers using Fargate model instead of the previous model where you would still need to create and manage VM instances to run the containers.

How does it work?

With the Jenkins ECS plugin you can configure a “Cloud” item that will launch all your agents on ECS Fargate, matching jobs to different container templates using labels. This means you can have unlimited agents with no machines to manage and just pay for what you use.

Some tips on the configuration:

  • Some options need to be configured, like subnet, security group and assign a public ip to the container in order to launch in Fargate.
  • Agents need to adhere to some predefined cpu and memory configurations. For instance for 1 vCPU you can only use 2GB to 8GB in 1GB increments.


Price per vCPU is $0.00001406 per second ($0.0506 per hour) and per GB memory is $0.00000353 per second ($0.0127 per hour).

If you compare the price with a m5.large instance (4 vCPU, 16 GB) that costs $0.192 per hour, it would cost you $0,4056 in Fargate, more than twice, ouch! You could build something similar and cheaper with Kubernetes using the cluster autoscaler given you can achieve a high utilization of the machines.

While I was writing this post someone already beat me to submit a PR to the ECS plugin to add the Fargate support.

Kubernetes Plugin for Jenkins 1.0

Includes support to get container logs from the pod, Kubernetes API auto configuration and lots of bug fixes

The full changelog:

  • containerLog step to get the logs of a container running in the agent pod JENKINS-46085 #195
  • Autoconfigure cloud if kubernetes url is not set #208
  • Change containerCap and instanceCap 0 to mean do not use JENKINS-45845 #199
  • Add environment variables to container from a secret JENKINS-39867 #162
  • Deprecate containerEnvVar for envVar and added secretEnvVar
  • Enable setting slaveConnectTimeout in podTemplate defined in pipeline #213
  • Read Jenkins URL from cloud configuration or KUBERNETES_JENKINS_URL env var #216
  • Make withEnv work inside a container JENKINS-46278 #204
  • Close resource leak, fix broken pipe error. Make number of concurrent requests to Kubernetes configurable JENKINS-40825 #182
  • Delete pods in the cloud namespace when pod namespace is not defined JENKINS-45910 #192
  • Use Util.replaceMacro instead of our custom replacement logic. Behavior change: when a var is not defined it is not replaced, ie. ${key1} or ${key2} or ${key3} -> value1 or value2 or ${key3} #198
  • Allow to create non-configurable instances programmatically #191
  • Do not cache kubernetes connection to reflect config changes and credential expiration JENKINS-39867 #189
  • Inherit podAnnotations when inheriting pod templates #209
  • Remove unneeded plugin dependencies, make pipeline-model-extensions optional #214

Speaking Trips on DevOps, Kubernetes, Jenkins

This 2nd half of the year speaking season is starting and you’ll find me speaking about DevOps, Kubernetes, Jenkins,… at

If you organize a conference and would like me to give a talk in 2018 you can find me @csanchez.

Screen Shot 2017-08-24 at 17.07.45.png

Kubernetes Plugin for Jenkins 0.12

Includes declarative pipeline support (note that you need Jenkins 2.66+ for it to work) and lots of bug fixes

The full changelog:

  • Add an experimental Declarative Agent extension for Kubernetes JENKINS-41758 #127
  • Implement Port mapping #165
  • Support idleMinutes field in pipeline #154
  • Add command liveness probe support #158
  • Add toggle for node usage mode #158
  • Add namespace support on PodTemplate.
  • Make PodTemplate optional within pipeline JENKINS-42315
  • Make Slave Jenkins connection timeout configurable #141
  • Fix durable pipeline PID NumberFormatException JENKINS-42048 #157
  • Don’t provision nodes if there are no PodTemplates set to usage mode Normal #171
  • Refactoring add/set methods in PodTemplate #173
  • Delete the build pod after we have finished with the template block #172
  • Default to use the kubernetes.default.svc.cluster.local endpoint
  • Do not print stack trace on ConnectException
  • Upgrade kubernetes client to 2.3.1 JENKINS-44189
  • Step namespace should have priority over anything else #161
  • Wait for pod to exist up to 60 seconds before erroring #155
  • Catch IOException on ContainerExecProc#kill
  • Do not print stack trace on connection exception
  • Restore random naming for pipeline managed pod templates.
  • Dir context is not honored by shell step JENKINS-40925 #146
  • Limit pod name to 63 characters, and change the randomly generated string #143
  • Fix workingDir inheritance error #136
  • Use name instead of label for the nesting stack #137
  • Exception in configure page when ‘Kubernetes URL’ isn’t filled JENKINS-45282 #174
  • kubectl temporary config file should work where Jenkins project contains spaces #178
  • Thread/connection leak #177

Running a JVM in a Container Without Getting Killed

No pun intended

The JDK 8u131 has backported a nice feature in JDK 9, which is the ability of the JVM to detect how much memory is available when running inside a Docker container.

I have talked multiple times about the problems running a JVM inside a container, how it will default in most cases to a max heap of 1/4 of the host memory, not the container.

For example in my machine

$ docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 444.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Wait, WAT? I set a container memory of 100MB and my JVM sets a max heap of 444M ? It is very likely that it is going to cause the Kernel to kill my JVM at some point.

Let’s try the JDK 8u131 with the experimental option -XX:+UseCGroupMemoryLimitForHeap

$ docker run -m 100MB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 44.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Ok this makes more sense, the JVM was able to detect the container has only 100MB and set the max heap to 44M.

Let’s try in a bigger container

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 228.00M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Mmm, now the container has 1GB but JVM is only using 228M as max heap. Can we optimize this even more, given that nothing else other than the JVM is running in the container? Yes we can!

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 910.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Using -XX:MaxRAMFraction we are telling the JVM to use available memory/MaxRAMFraction as max heap. Using -XX:MaxRAMFraction=1 we are using almost all the available memory as max heap.

UPDATE: follow up for Java 10+ at Running a JVM in a Container Without Getting Killed II

Docker Registry with Let’s Encrypt Certificate

A one-liner to run a SSL Docker registry generating a Let’s Encrypt certificate.

This command will create a registry proxying the Docker hub, caching the images in a registry volume.

LetsEncrypt certificate will be auto generated and stored in the host dir as letsencrypt.json. You could also use a Docker volume to store it.

In order for the certificate generation to work the registry needs to be accessible from the internet in port 443. After the certificate is generated that’s no longer needed.

docker run -d -p 443:5000 --name registry \
  -v `pwd`:/etc/docker/registry/ \
  -v registry:/var/lib/registry \
  -e REGISTRY_HTTP_TLS_LETSENCRYPT_CACHEFILE=/etc/docker/registry/letsencrypt.json \
  -e \

You can also create a config.yml in this dir and run the registry using the file instead of environment variables

version: 0.1
      cachefile: /etc/docker/registry/letsencrypt.json

Then run

docker run -d -p 443:5000 --name registry \
  -v `pwd`:/etc/docker/registry/ \
  -v registry:/var/lib/registry \

If you want to use this as a remote repository and not just for proxying, remove the proxy entry in the configuration

Running Docker containers as non root

Running containers as root is a bad practice, but many Docker images available in the Docker Hub have the user set to root by default, so what can we do about it?

TL;DR Use -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp for a typical Java image, plus any tool specific environment variable needed

Option 1 (build time):

Create a new derived image that creates a new user and changes the default to that one

FROM openjdk:8-jdk
RUN useradd --create-home -s /bin/bash user
WORKDIR /home/user
USER user

This is simple, but forces us to republish all these derived images, creating a maintenance nightmare.

Option 2 (runtime):

Use docker run -u option to choose what user to run the container as

docker run -ti --rm -u 1000 openjdk:8-jdk

This may work, but we can hit some issues, let’s see

$docker run -ti --rm -u 1000 openjdk:8-jdk git clone
fatal: could not create work tree dir 'docker'.: Permission denied

Well, we obviously don’t have permissions to write to the default workdir, let’s fix it using -w and a dir that is writable, for instance /tmp

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk git clone
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.
fatal: unable to look up current user in the passwd file: no such user


Git does not like being run as an user that does not exist, so we need to pick one of the existing users

UPDATE: git 2.6.5+ removes this requirement, so we can run as any user even if it is not in the passwd file. For previous versions setting the GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL environment variables also works.

$ docker run -ti --rm -u 1000 -w /tmp openjdk:8-jdk cat /etc/passwd
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false

The user nobody:65534 could be a good candidate, and as it is present in the default debian and alpine Docker images it will be present also in most images in the hub.

$ docker run -ti --rm -u 65534 -w /tmp openjdk:8-jdk \
git clone
Cloning into 'docker'...
remote: Counting objects: 1498, done.
remote: Total 1498 (delta 0), reused 0 (delta 0), pack-reused 1498
Receiving objects: 100% (1498/1498), 287.46 KiB | 0 bytes/s, done.
Resolving deltas: 100% (772/772), done.
Checking connectivity... done.

Ok, that worked! Now let’s try to run something else, like a maven build

$ docker run -ti --rm -u 65534 -w /tmp maven:3 \
bash -c "git clone && \
cd kubernetes-plugin && mvn package"
touch: cannot touch ‘/root/.m2/copy_reference_file.log’: Permission denied
Can not write to /root/.m2/copy_reference_file.log. Wrong volume permissions?

This means entering in the domain of each tool and checking how to configure it. The maven docker image instructs us to use MAVEN_CONFIG and pass -Duser.home otherwise we would get an error [ERROR] Could not create local repository at /nonexistent/.m2/repository -> [Help 1]

Here is the working solution

$ docker run -ti --rm -u 65534 -w /tmp -e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone && \
cd kubernetes-plugin && mvn -Duser.home=/tmp package"

Can we generalize this a bit for other Java apps? yes! By using the env var _JAVA_OPTIONS we can pass the user.home property to any Java app.

$ docker run -ti --rm -u 65534 -w /tmp -e _JAVA_OPTIONS=-Duser.home=/tmp \
-e MAVEN_CONFIG=/tmp maven:3 \
bash -c "git clone && \
cd kubernetes-plugin && mvn package"