Sending Kubernetes Logs to CloudWatch Logs using Fluentd

fluentd-logofluentd can send all the Kubernetes or EKS logs to CloudWatch Logs to have a centralized and unified view of all the logs from the cluster, both from the nodes and from each container stdout.

Installation

To send all nodes and container logs to CloudWatch, create a CloudWatch log group named kubernetes.

aws logs create-log-group --log-group-name kubernetes

Then install fluentd-cloudwatch helm chart. This will send logs from node, containers, etcd,… to CloudWatch as defined in the default fluentd chart config.

helm install --name fluentd incubator/fluentd-cloudwatch \
  --set awsRegion=us-east-1,rbac.create=true

Each node needs to have permissions to write to CloudWatch Logs, so either add the permission using IAM instance profiles or pass the awsRole if your are using kube2iam.

helm install --name fluentd incubator/fluentd-cloudwatch \
  --set awsRole=arn:aws:iam::123456789012:role/k8s-logs,awsRegion=us-east-1,rbac.create=true,extraVars[0]="{ name: FLUENT_UID, value: '0' }"<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>

The k8s-logs role policy is configured as

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "logs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}

Now you can go to CloudWatch and find your logs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s