Send Kubernetes cluster logs to AWS Elasticsearch

I had a similar problem. Below are the full details of how I got it working.

Setup:

• AWS ES instance accessible via a VPC.
• Using this yaml file as a template.
• k8s client version v1.9.2
• k8s server version v1.8.7

Host problem:

The main problem I had was with defining the environment variables correctly. For FLUENT_ELASTICSEARCH_HOST, I was including the https:// prefix on the host URL. Once I removed that, my connection problems went away.

Authentication:

There's no username or password configured for AWS ES. Per this discussion, I set the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD values to null.

Sample configuration:

Here's the full set of environment variables in my daemonset yaml file:

- name:  FLUENT_ELASTICSEARCH_HOST
  value: "vpc-MY-DOMAIN.REGION.es.amazonaws.com"
- name:  FLUENT_ELASTICSEARCH_PORT
  value: "443"
- name: FLUENT_ELASTICSEARCH_SCHEME
  value: "https"
- name: FLUENT_ELASTICSEARCH_USER
  value: null
- name: FLUENT_ELASTICSEARCH_PASSWORD
  value: null

Bonus: connecting to Kibana

Instead of setting up AWS Cognito, I created an nginx pod in my kubernetes cluster that I use as a proxy to reach Kibana. I use the kubectl port-foward command to reach the nginx server from my local machine.

Here's my nginx.conf:

server {
  listen 80;
  listen [::]:80;

  server_name MY-DOMAIN;

  location /_plugin/kibana {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com/_plugin/kibana;
  }
  location / {
      proxy_pass https://vpc-MY-DOMAIN.REGION.es.amazonaws.com;
  }
}

Once the nginx pod is deployed, I run this command:

kubectl port-forward POD_NAME 8888:80

Now the Kibana is accessible at http://localhost:8888/_plugin/kibana

I'm still having a timeout issue with the port-foward command and a problem with nginx caching the ES service IP (since that can change), but I'll update my response once I resolve those issues.