Kubernetes cluster on AWS EKS, Part 4: Setup Elastic Stack ( ElasticSearch / Kibana / APM Server ) on Kubernetes

This is a series on setting up Kubernetes clusters in Amazon EKS.

In this post, we will set up Elastic Stack on AWS EKS Cluster.

We will be setting up

  1. Elastic Stack (7.17)
  2. Nginx Ingress Controller (Shared)
  3. Certificates for all Ingress Services

ElasticStack installation will consist of

  1. Elastic Search
  2. Kibana
  3. APM Server

Note: The same setup will work for the latest version of Elastic which is 8.6.2 at the time of writing. APM Server does not work with version 8, as Elastic has deprecated APM in favor of ElasticFleet

Prequisites

  • AWS EKS Cluster
  • EBS CSI Driver Setup for PVC
  • Nginx Ingress Controller
  • Cert Manager

1. Elastic Search Installation

We will be using HELM Charts to install ElasticSearch.

The documentation can seem tricky, as Elastic Search is used for a wide range of use cases. There are 3 core components required for ElasticSearch to work

  • Master Node
  • Data Node
  • Client Node

For high-volume searches, it is recommended to install all 3 core components on different PODs. In our case, we are installing all 3 services on the same POD.

The roles a POD assumes can be customized using the roles variable in values.yaml

Let’s start the installation by adding the Elastic Helm Repo

helm repo add elastic https://helm.elastic.co
helm repo update

Download a copy of ElasticSearch Values

wget https://raw.githubusercontent.com/elastic/helm-charts/main/elasticsearch/values.yaml -O elastic-values.yaml

Update the following values in elastic-values.yaml

imageTag: "7.17.9"

# Increase based on the requirement. Following are the bare minimum limits for optimal performance
resources:
  requests:
    cpu: "1000m"
    memory: "1200Mi"
  limits:
    cpu: "1100m"
    memory: "1700Mi"

# Persistent Volume Claims
volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2

# We will attach the following service to an Ingress at a later step
service:
  enabled: true
  type: ClusterIP

Install ElasticSearch through HELM

helm install elasticsearch --values elastic-values.yaml  elastic/elasticsearch

Verify the installation

kubectl get pods

The output should be similar to

NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                   1/1     Running   0          6m
elasticsearch-master-1                   1/1     Running   0          6m
elasticsearch-master-2                   1/1     Running   0          6m

Add Ingress for ElasticSearch

Save the following in elastic-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-elasticsearch-ingress
  annotations:
   kubernetes.io/ingress.class: nginx
   cert-manager.io/cluster-issuer: letsencrypt-prod
   nginx.ingress.kubernetes.io/backend-protocol: HTTPS
   nginx.ingress.kubernetes.io/proxy-body-size: 20m

 spec:
   tls:
     - hosts:
          - elastic.example.com
       secretName: tls-elastic
rules:
- host: elastic.example.com
  http:
       paths:
       - path: /
         pathType: Prefix
         backend:
           service:
             name: elasticsearch-master
             port:
                number: 9200

Run the following to apply the ingress. Please ensure CNAME or A RECORD for the domain are set correctly to the Ingress LoadBalancer

kubectl apply -f elastic-ingress.yaml

2. Kibana Installation

Kibana sits on top of the Elastic Stack, providing search and data visualisation capabilities for data indexed.

Again, will be using HELM to install Kibana

Download a copy of Kibana Values

wget https://raw.githubusercontent.com/elastic/helm-charts/blob/main/kibana/values.yaml -O kibana-values.yaml

Update the following values in kibana-values.yaml

imageTag: "7.17.9"

# Following are bare minimum limits for optimal performance
resources:
  requests:
    cpu: "400m"
    memory: "700Mi"
  limits:
    cpu: "700m"
    memory: "1Gi"
 
service:
  enabled: true
  type: ClusterIP

Install Kibana through HELM

helm install kibana --values kibana-values.yaml  elastic/elasticsearch

Verify the installation

kubectl get pods

The output should be similar to

NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                   1/1     Running   0          16m
elasticsearch-master-1                   1/1     Running   0          16m
elasticsearch-master-2                   1/1     Running   0          16m
kibana-kibana-6978bfcbbd-nchll           1/1     Running   0          4m

Add Ingress for Kibana

Save the following in kibana-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-kibana-ingress
  annotations:
   kubernetes.io/ingress.class: nginx
   cert-manager.io/cluster-issuer: letsencrypt-prod
   nginx.ingress.kubernetes.io/backend-protocol: HTTPS


 spec:
   tls:
     - hosts:
          - kibana.example.com
       secretName: tls-kibana
rules:
- host: kibana.example.com
  http:
       paths:
       - path: /
         pathType: Prefix
         backend:
           service:
             name: kibana-kibana
             port:
                number: 5601

Kibana dashboard is accessible at kibana.example.com

Username: elastic

Password: Stored in Kubernetes secret > elasticsearch-master-credential

3. APM Server Installation

APM server pipes, logs received from APM agents like Elastic APM agent for Ruby and pipes them to ElasticSearch. This data is readily indexed and is used in Kibana for visualizations.

Again, will be using HELM to install APM Server

Download a copy of APM Server Values

wget https://raw.githubusercontent.com/elastic/helm-charts/main/apm-server/values.yaml -O apm-values.yaml

Update the following values in apm.yaml

imageTag: "7.17.9"

apmConfig:
  apm-server.yml: |
    apm-server:
      host: "0.0.0.0:8200"
      secret_token: "YOUR_SECRET_TOKEN"
      write_timeout: 30s
      rum.enabled: true
      rum.event_rate.limit: 300
      rum.event_rate.lru_size: 1000
      rum.allow_origins: ['*']
    queue: {}
    output.elasticsearch:
      hosts: ["http://elasticsearch-master:9200"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"

imageTag: "7.17.9"

# Bare minimum limits for optimal performance
resources:
  requests:
    cpu: "100m"
    memory: "100Mi"
  limits:
    cpu: "600m"
    memory: "512Mi"
 
service:
  type: ClusterIP

Install APM Server through HELM

helm install apm-server --values apm-values.yaml  elastic/elasticsearch

Verify the installation

kubectl get pods

The output should be similar to

NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                   1/1     Running   0          26m
elasticsearch-master-1                   1/1     Running   0          26m
elasticsearch-master-2                   1/1     Running   0          26m
kibana-kibana-6978bfcbbd-nchll           1/1     Running   0          14m
apm-server-apm-server-86c9cb8d5c-scr9x   1/1     Running   0          6m

Add Ingress for APM Server

Save the following in apm-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-apm-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
   cert-manager.io/cluster-issuer: letsencrypt-prod
   nginx.ingress.kubernetes.io/backend-protocol: HTTPS

 spec:
   tls:
     - hosts:
          - apm.example.com
       secretName: tls-apm
rules:
- host: apm.example.com
  http:
       paths:
       - path: /
         pathType: Prefix
         backend:
           service:
             name: apm-server-apm-server
             port:
                number: 8200                

Open apm.example.com and you should see something like

  
{
	"build_date": "2022-04-19T06:13:49Z",
	"build_sha": "d3dc31c5db912b297835f934292218c2af5ea7ba",
	"publish_ready": true,
	"version": "7.17.9"
}

publish_ready:true states the server is ready to receive data.

Once, the Agent is connected you can have ready-made visualization under Kibana > APM like

Need help on your Ruby on Rails or React project?

Join Our Newsletter