Google Cloud Next Recap

google-next-logoSeveral interesting announcements from last week Google Next conference.

Knative, a new OSS project built by Google, Red Hat, IBM,… to build, deploy, and manage modern serverless workloads on Kubernetes. Built upon Istio, with 1.0 coming soon and managed Istio on GCP. It includes a build primitive to manage source to kubernetes flows, that can be used independently. Maybe it is the new standard to define sources and builds in Kubernetes. Read more from Mark Chmarny.

GKE on premise, a Google-configured version of Kubernetes with multi-cluster management, running on top of VMware’s vSphere.

Another Kubernetes related mention was the gVisor pod sandbox, with experimental support for Kubernetes, to allow running sandboxed containers in a Kubernetes cluster. Very interesting for multi-tenant clusters and docker image builds.

Cloud Functions are now Generally Available, and more serverless features are launched:

Serverless containers allow you to run container-based workloads in a fully managed environment and still only pay for what you use. Sign up for an early preview of serverless containers on Cloud Functions to run your own containerized functions on GCP with all the benefits of serverless.

A new GKE serverless add-on lets you run serverless workloads on Kubernetes Engine with a one-step deploy. You can go from source to containers instantaneously, auto-scale your stateless container-based workloads, and even scale down to zero.

Cloud Build, a fully-managed CI/CD platform that lets you build and test applications in the cloud. With an interesting approach where all the pipeline steps are containers themselves so it is reasonably easy to extend. It integrates with GitHub for repos with a Dockerfile (let’s see if it lasts long after Microsoft acquisition).

Other interesting announcements include:

  • Edge TPU, a tiny ASIC chip designed to run TensorFlow Lite ML models at the edge.
  • Shielded VMs – untampered virtual machines

  • Titan Security Key, a FIDO security key with firmware developed by Google. Google security was giving away at the conference both NFC and bluetooth keys, a good replacement for the yubikeys specially for mobile devices.

Sending Kubernetes Logs to CloudWatch Logs using Fluentd

fluentd-logofluentd can send all the Kubernetes or EKS logs to CloudWatch Logs to have a centralized and unified view of all the logs from the cluster, both from the nodes and from each container stdout.

Installation

To send all nodes and container logs to CloudWatch, create a CloudWatch log group named kubernetes.

aws logs create-log-group --log-group-name kubernetes

Then install fluentd-cloudwatch helm chart. This will send logs from node, containers, etcd,… to CloudWatch as defined in the default fluentd chart config.

helm install --name fluentd incubator/fluentd-cloudwatch \
  --set awsRegion=us-east-1,rbac.create=true

Each node needs to have permissions to write to CloudWatch Logs, so either add the permission using IAM instance profiles or pass the awsRole if your are using kube2iam.

helm install --name fluentd incubator/fluentd-cloudwatch \
  --set awsRole=arn:aws:iam::123456789012:role/k8s-logs,awsRegion=us-east-1,rbac.create=true,extraVars[0]="{ name: FLUENT_UID, value: '0' }"<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>

The k8s-logs role policy is configured as

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "logs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

Now you can go to CloudWatch and find your logs.

Installing kube2iam in AWS Kubernetes Kops Cluster

kuberneteskube2iam allows a Kubernetes cluster in AWS to use different IAM roles for each pod, and prevents pods from accessing EC2 instance IAM roles.

Installation

Edit your kops cluster with kops edit cluster to allow nodes to assume different roles, changing the account id 123456789012 to yours

 spec:
  additionalPolicies:
    nodes: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "sts:AssumeRole"
          ],
          "Resource": [
            "arn:aws:iam::123456789012:role/k8s-*"
          ]
        }
      ]

Install kube2iam using the helm chart

helm install stable/kube2iam --namespace kube-system --name my-release \
  --set=extraArgs.base-role-arn=arn:aws:iam::123456789012:role/,extraArgs.default-role=kube2iam-default,host.iptables=true,host.interface=cbr0,rbac.create=true

A curl to the metadata server from a new pod should return kube2iam

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
kube2iam

Role configuration

Create the roles that the pods can assume. They must start with k8s- (see the wildcard we set in the Resource above) and contain a trust relationship to the node pool role.

For instance, to allow access to the S3 bucket mybucket from a pod, create a role k8s-s3.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "s3bucketActions",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::mybucket",
       }
    ]
}

Then edit the trust relationship of the role to allow the node role (the role created by Kops for the Auto Scaling Goup) to assume this role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/nodes.example.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Test it by launching a pod with the right annotations

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
  labels:
    name: aws-cli
  annotations:
    iam.amazonaws.com/role: arn:aws:iam::123456789012:role/k8s-s3
spec:
  containers:
  - image: fstab/aws-cli
    command:
      - "/home/aws/aws/env/bin/aws"
      - "s3"
      - "ls"
      - "some-bucket"
    name: aws-cli

Securing namespaces

kube2iam supports namespace restrictions so users can still launch pods but are limited to a predefined set of IAM roles that can assume.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    iam.amazonaws.com/allowed-roles: |
      ["my-custom-path/*"]
  name: default

A similar setup should also work with AWS EKS.

Kubernetes Plugin for Jenkins 1.7.1 Security Release

A minor security issue has been found and fixed in 1.7.1

  • Do not print credentials in build output or logs. Only affects certain pipeline steps like withDockerRegistrysh step is not affected

Other interesting new feature is the support of multiple containers in declarative pipeline #306 JENKINS-48135

</pre>
<pre>pipeline {
  agent {
    kubernetes {
      label 'mypod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: maven
    image: maven:alpine
    command:
    - cat
    tty: true
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
    }
  }
  stages {
    stage('Run maven') {
      steps {
        container('maven') {
          sh 'mvn -version'
        }
        container('busybox') {
          sh '/bin/busybox'
        }
      }
    }
  }
}</pre>
<pre>

1.7.1

  • Do not print credentials in build output or logs. Only affects certain pipeline steps like withDockerRegistrysh step is not affected SECURITY-883

1.7.0

  • Add option to apply caps only on alive pods #252
  • Add idleMinutes to pod template in declarative pipeline #336 JENKINS-51569

1.6.4

  • Use Jackson and Apache HttpComponents Client libraries from API plugins #333 JENKINS-51582

1.6.3

1.6.2

  • Transfer any master proxy related envs that the remoting jar uses to the pod templates with addMasterProxyEnvVarsoption #321

1.6.1

  • Some fields are not inherited from parent template (InheritFrom, InstanceCap, SlaveConnectTimeout, IdleMinutes, ActiveDeadlineSeconds, ServiceAccount, CustomWorkspaceVolumeEnabled) #319

1.6.0

  • Support multiple containers in declarative pipeline #306 JENKINS-48135
  • Expose pod configuration via yaml to UI and merge tolerations when inheriting #311
  • Resolve NPE merging yaml when resource requests/limits are not set #310
  • Do not pass arguments to jnlp container #315 JENKINS-50913

1.5.2

1.5.1

You can find the full changelog in GitHub.

Kubernetes Plugin for Jenkins 1.5

15 releases have gone by in 7 months since 1.0 last September

Some interesting new features since 1.0 and a lot of bugfixes and overall stability improvements. For instance now you can use yaml to define the Pod that will be used for your job:

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
) {
    node (label) {
      container('busybox') {
        sh "hostname"
      }
    }
}

 

You can use readFile step to load the yaml from a file in your git repo.

  • Allow creating Pod templates from yaml. This allows setting all possible fields in Kubernetes API using yaml JENKINS-50282 #275
  • Support passing kubeconfig file as credentials using secretFile credentials JENKINS-49817 #294

You can find the full changelog in GitHub.

Kubernetes Plugin for Jenkins 1.0

Includes support to get container logs from the pod, Kubernetes API auto configuration and lots of bug fixes

The full changelog:

  • containerLog step to get the logs of a container running in the agent pod JENKINS-46085 #195
  • Autoconfigure cloud if kubernetes url is not set #208
  • Change containerCap and instanceCap 0 to mean do not use JENKINS-45845 #199
  • Add environment variables to container from a secret JENKINS-39867 #162
  • Deprecate containerEnvVar for envVar and added secretEnvVar
  • Enable setting slaveConnectTimeout in podTemplate defined in pipeline #213
  • Read Jenkins URL from cloud configuration or KUBERNETES_JENKINS_URL env var #216
  • Make withEnv work inside a container JENKINS-46278 #204
  • Close resource leak, fix broken pipe error. Make number of concurrent requests to Kubernetes configurable JENKINS-40825 #182
  • Delete pods in the cloud namespace when pod namespace is not defined JENKINS-45910 #192
  • Use Util.replaceMacro instead of our custom replacement logic. Behavior change: when a var is not defined it is not replaced, ie. ${key1} or ${key2} or ${key3} -> value1 or value2 or ${key3} #198
  • Allow to create non-configurable instances programmatically #191
  • Do not cache kubernetes connection to reflect config changes and credential expiration JENKINS-39867 #189
  • Inherit podAnnotations when inheriting pod templates #209
  • Remove unneeded plugin dependencies, make pipeline-model-extensions optional #214

Speaking Trips on DevOps, Kubernetes, Jenkins

This 2nd half of the year speaking season is starting and you’ll find me speaking about DevOps, Kubernetes, Jenkins,… at

If you organize a conference and would like me to give a talk in 2018 you can find me @csanchez.

Screen Shot 2017-08-24 at 17.07.45.png