Centralized logging for kubernetes with fluentd and elasticsearch

I had to setup a centralized monitoring for our production and staging applications. In my company, we saw a need to have an alternative solution (introduction of pricing, etc.) to the google stackdrive logger in place.

Having some bad experience with running Logstash, I decided to go with EFK (elasticsearch, fluentd, and kibana). Plus. I could reuse the existing fluentd deployment definition in the kubernetes github repo, so it was easy to get something running in short time.

To save time and avoid fine-tuning kubernetes pods (adding a pool of machines capable running elasticsearch), I decided to run the elasticsearch on a dedicated virtual machine. With terraform and docker-compose, I could recreate the elasticsearch in few minutes.

On Google Cloud Platform every virtual machine gets DNS entry:
<MACHINE_NAME>.c.<PROJECT_NAME>.internal.

So, my cental machine got the following name: elasticsearch.c.my-project.internal

The kubernetes part was really easy, thanks to the service discovery.

1. Setup an ExternalService with name: elasticsearch-logging that points to the elasticsearch instance:

apiVersion: v1
 kind: Service
 metadata:
   name: elasticsearch-logging
   namespace: kube-system
   labels:
     k8s-app: elasticsearch
     kubernetes.io/name: "elasticsearch"
 spec:
   type: ExternalName
   externalName: elasticsearch.c.my-project.internal
   ports:
     - port: 9200
       targetPort: 9200

Now, let’s setup the fluentd agents on every kubernetes node in my cluster.

2. Create the fluentd agent configuration (please notice, I commented out the section for gathering kubernetes internal logs) based on the kubernetes repo:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
data:
  td-agent.conf: |
    <match fluent.**>
      type null
    </match>
    
    # Example:
    # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
    <source>
      type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag kubernetes.*
      format json
      read_from_head true
    </source>
    
    # Example:
    # 2015-12-21 23:17:22,066 [salt.state       ][INFO    ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
    <source>
      type tail
      format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
      time_format %Y-%m-%d %H:%M:%S
      path /var/log/salt/minion
      pos_file /var/log/es-salt.pos
      tag salt
    </source>
    
    # Example:
    # Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
    <source>
      type tail
      format syslog
      path /var/log/startupscript.log
      pos_file /var/log/es-startupscript.log.pos
      tag startupscript
    </source>
    
    # Examples:
    # time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
    # time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
    <source>
      type tail
      format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      path /var/log/docker.log
      pos_file /var/log/es-docker.log.pos
      tag docker
    </source>
    
    # Example:
    # 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  # Not parsing this, because it doesn't have anything particularly useful to
    #  # parse out of it (like severities).
    #  format none
    #  path /var/log/etcd.log
    #  pos_file /var/log/es-etcd.log.pos
    #  tag etcd
    #</source>
    
    # Multi-line parsing is required for all the kube logs because very large log
    # statements, such as those that include entire object bodies, get split into
    # multiple lines by glog.
    
    # Example:
    # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  format multiline
    #  format_firstline /^\w\d{4}/
    #  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    #  time_format %m%d %H:%M:%S.%N
    #  path /var/log/kubelet.log
    #  pos_file /var/log/es-kubelet.log.pos
    #  tag kubelet
    #</source>
    
    # Example:
    # I0204 07:00:19.604280       5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  format multiline
    #  format_firstline /^\w\d{4}/
    #  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    #  time_format %m%d %H:%M:%S.%N
    #  path /var/log/kube-apiserver.log
    #  pos_file /var/log/es-kube-apiserver.log.pos
    #  tag kube-apiserver
    #</source>
    
    # Example:
    # I0204 06:55:31.872680       5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  format multiline
    #  format_firstline /^\w\d{4}/
    #  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    #  time_format %m%d %H:%M:%S.%N
    #  path /var/log/kube-controller-manager.log
    #  pos_file /var/log/es-kube-controller-manager.log.pos
    #  tag kube-controller-manager
    #</source>
    
    # Example:
    # W0204 06:49:18.239674       7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  format multiline
    #  format_firstline /^\w\d{4}/
    #  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    #  time_format %m%d %H:%M:%S.%N
    #  path /var/log/kube-scheduler.log
    #  pos_file /var/log/es-kube-scheduler.log.pos
    #  tag kube-scheduler
    #</source>
    
    <filter kubernetes.**>
      type kubernetes_metadata
    </filter>
    
    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    # NOTICE: :Wojtek: I do not want to get the kubelet logs, only pod logs.
    #<source>
    #  type tail
    #  format multiline
    #  multiline_flush_interval 5s
    #  format_firstline /^\w\d{4}/
    #  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    #  time_format %m%d %H:%M:%S.%N
    #  path /var/log/glbc.log
    #  pos_file /var/log/es-glbc.log.pos
    #  tag glbc
    #</source>
    
    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    <source>
      type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/cluster-autoscaler.log
      pos_file /var/log/es-cluster-autoscaler.log.pos
      tag cluster-autoscaler
    </source>
    
    <match **>
       type elasticsearch
       log_level info
       include_tag_key true
       host elasticsearch-logging
       port 9200
       logstash_format true
       # Set the chunk limit the same as for fluentd-gcp.
       buffer_chunk_limit 2M
       # Cap buffer memory usage to 2MiB/chunk * 32 chunks = 64 MiB
       buffer_queue_limit 32
       flush_interval 5s
       # Never wait longer than 5 minutes between retries.
       max_retry_wait 30
       # Disable the limit on the number of retries (retry forever).
       disable_retry_limit
       # Use multiple threads for processing.
       num_threads 8
    </match>

3. Deploy a fluentd-elasticsearch as a DeamonSet. fluentd-elasticsearch will automatically connect to any service with name `elasticsearch-logging` (based on a fluentd-elasticsearch deployment defintion) :

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    tier: monitoring
    app: fluentd-logging
    k8s-app: fluentd-logging
spec:
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      containers:
        - name: fluentd-elasticsearch
          image: gcr.io/google_containers/fluentd-elasticsearch:1.19
          volumeMounts:
          - name: varlog
            mountPath: /var/log
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
            readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Use `kubectl logs fluentd-elasticsearch-...` to check whether you were able to connect to the elasticsearach instance.

4. Now, you can open the kibana and see the logs.

 

wb