Kubernetes Plugin for Jenkins 1.7.1 Security Release

A minor security issue has been found and fixed in 1.7.1

  • Do not print credentials in build output or logs. Only affects certain pipeline steps like withDockerRegistrysh step is not affected

Other interesting new feature is the support of multiple containers in declarative pipeline #306 JENKINS-48135

</pre>
<pre>pipeline {
  agent {
    kubernetes {
      label 'mypod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: maven
    image: maven:alpine
    command:
    - cat
    tty: true
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
    }
  }
  stages {
    stage('Run maven') {
      steps {
        container('maven') {
          sh 'mvn -version'
        }
        container('busybox') {
          sh '/bin/busybox'
        }
      }
    }
  }
}</pre>
<pre>

1.7.1

  • Do not print credentials in build output or logs. Only affects certain pipeline steps like withDockerRegistrysh step is not affected SECURITY-883

1.7.0

  • Add option to apply caps only on alive pods #252
  • Add idleMinutes to pod template in declarative pipeline #336 JENKINS-51569

1.6.4

  • Use Jackson and Apache HttpComponents Client libraries from API plugins #333 JENKINS-51582

1.6.3

1.6.2

  • Transfer any master proxy related envs that the remoting jar uses to the pod templates with addMasterProxyEnvVarsoption #321

1.6.1

  • Some fields are not inherited from parent template (InheritFrom, InstanceCap, SlaveConnectTimeout, IdleMinutes, ActiveDeadlineSeconds, ServiceAccount, CustomWorkspaceVolumeEnabled) #319

1.6.0

  • Support multiple containers in declarative pipeline #306 JENKINS-48135
  • Expose pod configuration via yaml to UI and merge tolerations when inheriting #311
  • Resolve NPE merging yaml when resource requests/limits are not set #310
  • Do not pass arguments to jnlp container #315 JENKINS-50913

1.5.2

1.5.1

You can find the full changelog in GitHub.

Kubernetes Plugin for Jenkins 1.5

15 releases have gone by in 7 months since 1.0 last September

Some interesting new features since 1.0 and a lot of bugfixes and overall stability improvements. For instance now you can use yaml to define the Pod that will be used for your job:

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
) {
    node (label) {
      container('busybox') {
        sh "hostname"
      }
    }
}

 

You can use readFile step to load the yaml from a file in your git repo.

  • Allow creating Pod templates from yaml. This allows setting all possible fields in Kubernetes API using yaml JENKINS-50282 #275
  • Support passing kubeconfig file as credentials using secretFile credentials JENKINS-49817 #294

You can find the full changelog in GitHub.

Serverless CI/CD with AWS ECS Fargate

Amazon AWS has recently launched ECS Fargate to “run containers without having to manage servers or clusters”.

So this got me interested enough to patch the Jenkins ECS plugin to run Jenkins agents as containers using Fargate model instead of the previous model where you would still need to create and manage VM instances to run the containers.

How does it work?

With the Jenkins ECS plugin you can configure a “Cloud” item that will launch all your agents on ECS Fargate, matching jobs to different container templates using labels. This means you can have unlimited agents with no machines to manage and just pay for what you use.

Some tips on the configuration:

  • Some options need to be configured, like subnet, security group and assign a public ip to the container in order to launch in Fargate.
  • Agents need to adhere to some predefined cpu and memory configurations. For instance for 1 vCPU you can only use 2GB to 8GB in 1GB increments.

Pricing

Price per vCPU is $0.00001406 per second ($0.0506 per hour) and per GB memory is $0.00000353 per second ($0.0127 per hour).

If you compare the price with a m5.large instance (4 vCPU, 16 GB) that costs $0.192 per hour, it would cost you $0,4056 in Fargate, more than twice, ouch! You could build something similar and cheaper with Kubernetes using the cluster autoscaler given you can achieve a high utilization of the machines.

While I was writing this post someone already beat me to submit a PR to the ECS plugin to add the Fargate support.

Kubernetes Plugin for Jenkins 1.0

Includes support to get container logs from the pod, Kubernetes API auto configuration and lots of bug fixes

The full changelog:

  • containerLog step to get the logs of a container running in the agent pod JENKINS-46085 #195
  • Autoconfigure cloud if kubernetes url is not set #208
  • Change containerCap and instanceCap 0 to mean do not use JENKINS-45845 #199
  • Add environment variables to container from a secret JENKINS-39867 #162
  • Deprecate containerEnvVar for envVar and added secretEnvVar
  • Enable setting slaveConnectTimeout in podTemplate defined in pipeline #213
  • Read Jenkins URL from cloud configuration or KUBERNETES_JENKINS_URL env var #216
  • Make withEnv work inside a container JENKINS-46278 #204
  • Close resource leak, fix broken pipe error. Make number of concurrent requests to Kubernetes configurable JENKINS-40825 #182
  • Delete pods in the cloud namespace when pod namespace is not defined JENKINS-45910 #192
  • Use Util.replaceMacro instead of our custom replacement logic. Behavior change: when a var is not defined it is not replaced, ie. ${key1} or ${key2} or ${key3} -> value1 or value2 or ${key3} #198
  • Allow to create non-configurable instances programmatically #191
  • Do not cache kubernetes connection to reflect config changes and credential expiration JENKINS-39867 #189
  • Inherit podAnnotations when inheriting pod templates #209
  • Remove unneeded plugin dependencies, make pipeline-model-extensions optional #214

Speaking Trips on DevOps, Kubernetes, Jenkins

This 2nd half of the year speaking season is starting and you’ll find me speaking about DevOps, Kubernetes, Jenkins,… at

If you organize a conference and would like me to give a talk in 2018 you can find me @csanchez.

Screen Shot 2017-08-24 at 17.07.45.png

Kubernetes Plugin for Jenkins 0.12

Includes declarative pipeline support (note that you need Jenkins 2.66+ for it to work) and lots of bug fixes

The full changelog:

  • Add an experimental Declarative Agent extension for Kubernetes JENKINS-41758 #127
  • Implement Port mapping #165
  • Support idleMinutes field in pipeline #154
  • Add command liveness probe support #158
  • Add toggle for node usage mode #158
  • Add namespace support on PodTemplate.
  • Make PodTemplate optional within pipeline JENKINS-42315
  • Make Slave Jenkins connection timeout configurable #141
  • Fix durable pipeline PID NumberFormatException JENKINS-42048 #157
  • Don’t provision nodes if there are no PodTemplates set to usage mode Normal #171
  • Refactoring add/set methods in PodTemplate #173
  • Delete the build pod after we have finished with the template block #172
  • Default to use the kubernetes.default.svc.cluster.local endpoint
  • Do not print stack trace on ConnectException
  • Upgrade kubernetes client to 2.3.1 JENKINS-44189
  • Step namespace should have priority over anything else #161
  • Wait for pod to exist up to 60 seconds before erroring #155
  • Catch IOException on ContainerExecProc#kill
  • Do not print stack trace on connection exception
  • Restore random naming for pipeline managed pod templates.
  • Dir context is not honored by shell step JENKINS-40925 #146
  • Limit pod name to 63 characters, and change the randomly generated string #143
  • Fix workingDir inheritance error #136
  • Use name instead of label for the nesting stack #137
  • Exception in configure page when ‘Kubernetes URL’ isn’t filled JENKINS-45282 #174
  • kubectl temporary config file should work where Jenkins project contains spaces #178
  • Thread/connection leak #177

Running a JVM in a Container Without Getting Killed

No pun intended

The JDK 8u131 has backported a nice feature in JDK 9, which is the ability of the JVM to detect how much memory is available when running inside a Docker container.

I have talked multiple times about the problems running a JVM inside a container, how it will default in most cases to a max heap of 1/4 of the host memory, not the container.

For example in my machine

$ docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 444.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Wait, WAT? I set a container memory of 100MB and my JVM sets a max heap of 444M ? It is very likely that it is going to cause the Kernel to kill my JVM at some point.

Let’s try the JDK 8u131 with the experimental option -XX:+UseCGroupMemoryLimitForHeap

$ docker run -m 100MB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 44.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Ok this makes more sense, the JVM was able to detect the container has only 100MB and set the max heap to 44M.

Let’s try in a bigger container

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 228.00M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Mmm, now the container has 1GB but JVM is only using 228M as max heap. Can we optimize this even more, given that nothing else other than the JVM is running in the container? Yes we can!

$ docker run -m 1GB openjdk:8u131 java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 910.50M
    Ergonomics Machine Class: server
    Using VM: OpenJDK 64-Bit Server VM

Using -XX:MaxRAMFraction we are telling the JVM to use available memory/MaxRAMFraction as max heap. Using -XX:MaxRAMFraction=1 we are using almost all the available memory as max heap.

UPDATE: follow up for Java 10+ at Running a JVM in a Container Without Getting Killed II