Tutorial: Add logging and monitoring to your cluster with Elasticsearch and Kibana

A critical part of any application deployment is monitoring by means of log analysis. Kubernetes lets you collect and aggregate logs across your cluster, so that you can monitor your entire cluster from a single dashboard.

One popular open source solution combines log collection and aggregation with Fluentd, log analytics with Elasticsearch, and data visualization with Kibana (EFK for short).

This tutorial shows how to add Elasticsearch logging and Kibana monitoring to a Kubernetes cluster running on AWS. It takes advantage of AWS services for Elasticsearch and Kibana, so that you install only Fluentd on your cluster.

For more information about Elasticsearch and Kibana, see the Elastic website. For Fluentd, see the Fluentd website.

Architecture and decisions

To add logging and monitoring to a Kubernetes Quick Start cluster on AWS, we add the following elements to the cluster:

  • Elasticsearch: as provided by AWS’s hosted Elasticsearch service.
  • Fluentd: installed on your cluster by applying a Kubernetes daemonset.
  • Kibana: as provided by an AWS plugin. You do not need to create your own Kibana instance.

Prerequisites

This tutorial assumes the following:

  • You have a local Linux/Unix environment (like a MacBook)
  • You have successfully created the CloudFormation stacks from the AWS Kubernetes Quick Start (or followed the CLI walkthrough if you prefer)
  • You have copied the resulting kubeconfig file to your local machine, and can successfully run kubectl commands against the cluster (see Step 4 of the setup guide)

Try running kubectl get nodes. Your output should look similar to this:

NAME               STATUS         AGE
ip-10-0-0-0     Ready,master      1h
ip-172-172-172-172 Ready          1h
ip-192-192-192-192 Ready          1h

If it is, your cluster is ready to go! If the output looks substantially different, try running through the setup walkthrough again.

Note:Make sure to check the versions of both your API server and your kubectl client. When you install Fluentd, YAML validation is performed client-side.

1. Create Elasticsearch domain on AWS

You can set up an AWS Elasticsearch domain by using either the Amazon Elasticsearch console UI or the AWS CLI. For details, see the AWS Elasticsearch documentation.

Note:To test or experiment with this approach, you can choose an Instance type of t2.small.elasticsearch (Free tier eligible).

Set the domain access policy to Allow access to the domain from specific IP(s), and specify the NATEIP of your CloudFormation stack as the value of the aws:SourceIp field.

Note:You can find the NATEIP for your stack on the Resources tab of your CloudFormation page. Or you can run the following command:
STACK=varMyStack

aws cloudformation describe-stack-resources --stack-name $STACK --logical-resource-id NATEIP

In the JSON document that’s returned, look for the value of the PhysicalResourceId field. This value is the NATEIP of your stack.

According to Amazon, a new domain can take up to ten minutes to initialize.

You can check the status of your Elasticsearch domain by running the following command:

DOMAINNAME=varMyElasticDomainName

aws es describe-elasticsearch-domain --domain-name $DOMAINNAME

In the JSON document that’s returned, look for the key/value pair "Created": true. Note also that the endpoint value in this document is your Elasticsearch host, or URL. You need this value to configure your Fluentd installation.

You can also view the domain configuration status in the AWS dashboard. The value should be Active, and the endpoint value is displayed.

2. Install Fluentd on Kubernetes cluster

You install Fluentd on your Kubernetes cluster by creating a daemonset with the appropriate YAML file. Two files are available at https://github.com/heptio/fluentd-yaml, one for Kubernetes version 1.5, and one for version 1.6.

Download the YAML file that corresponds to your version of Kubernetes, and then edit the file to provide the value of the FLUENT_ELASTICSEARCH_HOST field (varMyElasticDomain). This value is your AWS Elasticsearch URL.

spec:
  serviceAccountName: fluentd-service-account
  tolerations:
  - key: node-role.kubernetes.io/master
    operator: Exists
    effect: NoSchedule
  containers:
  - name: fluentd
    image: quay.io/fluent/fluentd-kubernetes-daemonset
    env:
      - name:  FLUENT_ELASTICSEARCH_HOST
        value: "<varMyElasticDomain>"
      - name:  FLUENT_ELASTICSEARCH_PORT
        value: "80"

After you edit the YAML file, run one of the following commands:

If your Kubernetes cluster is on version 1.6 with RBAC enabled (default):

kubectl apply -f fluentd-1-6.yaml

If your Kubernetes cluster is on version 1.5 without RBAC enabled (default):

kubectl apply -f fluentd-1-5.yaml

When this command succeeds, you see messages like the following:

daemonset "fluentd" configured
configmap "fluentd-configmap" created
clusterrolebinding "fluentd-service-account" configured
clusterrole "fluentd-service-account" configured
serviceaccount "fluentd-service-account" configured
Note:If you disable RBAC in version 1.6, see the Appendix for how to edit the YAML to avoid validation errors.

3. Connect to Kibana dashboard

To view your Kibana dashboard locally, you create an SSH proxy tunnel through your bastion host to your internal cluster and expose the Elasticsearch server over localhost:8080. This approach is required because of permissions settings on the AWS stack.

Run the following command:

SSH_KEY="path/to/varMyKey.pem"; ssh -i $SSH_KEY -L8080:<varMyElasticDomain>:80 -o ProxyCommand="ssh -i \"${SSH_KEY}\" ubuntu@<varMyBastionHostIp> nc %h %p" ubuntu@<varMyMasterNodeIp>
Note:Port 8080 must be available on localhost. If it is already assigned, you can change the port assignment in the command.

To get the value of <varMyElasticDomain>, run this command:

DOMAINNAME=varMyElasticDomainName

aws es describe-elasticsearch-domain --query 'DomainStatus.Endpoint' --output text --domain-name $DOMAINNAME

To get the value of <varMyBastionHostIp>, run this command:

STACK=varMyStack

aws cloudformation describe-stacks --query 'Stacks[*].Outputs[?OutputKey == `BastionHostPublicIp`].OutputValue' --output text --stack-name $STACK

To get the value of <varMyMasterNodeIp>, run this command:

STACK=varMyStack

aws cloudformation describe-stacks --query 'Stacks[*].Outputs[?OutputKey == `MasterPrivateIp`].OutputValue' --output text --stack-name $STACK

You can now view your Kibana dashboard at

localhost:8080/_plugin/kibana

When you visit this URL for the first time, you see a page that asks you to configure an index pattern. Select Index contains time-based events, choose the default Index name or pattern (logstash-*) and Time-field value (@timestamp), and click Create. You’ll see a view that includes the raw log data:


Kibana Discover page that displays log entries by timestamp in chart and raw form

For more information, see the Kubernetes documentation on logging. You can filter your logs with search queries, and you can configure the visualizations that you need for your specific monitoring requirements. For details, see the Kibana documentation. For example, the following image shows the count of log events by pod name.


Kibana Visualize page that displays count of log events by pod name

Appendix

If you disable role-based access control (RBAC) in version 1.6, you might want to remove the following lines from fluentd-1-6.yaml to avoid validation errors:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluentd-service-account
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-service-account
subjects:
- kind: ServiceAccount
  name: fluentd-service-account
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-service-account
  namespace: kube-system
rules:
  - apiGroups: ["*"]
    resources:
      - pods
      - namespaces
    verbs:
      - get
      - watch
      - list