How to reduce high CPU usage with Elasticsearch

user_1839104 0
Viewed 6694

I am using Elasticsearch 6.8.10 with Spring Boot 2.2.7 and Spring Data Elasticsearch.

I have streams of stats and trends data being stored in Kafka topics. These topics are read using Spring Kafka and stored into MongoDB and Elasticsearch for analysis and reporting. The problem I am having is that when the queues are being processed and the data is written to Elasticsearch, Elasticsearch CPU consumption is continuously around 250%. This leads to sporadic timeout errors across the application. I understand that indexing is an intensive operation but I am trying to understand what I can do to reduce the CPU usage.

Data:

  • Approx stats queue items (1.2M)
  • Stats document size (220 bytes)

The VM config details are:

  • 4 CPU, 16GB Memory, 20GB disk (SSD)
  • Running on a VM in Google Cloud Platform.
  • VM only used for Elasticsearch

The Docker Elasticsearch config details:

  • I am using single node (for the moment)
version: '2.4'
services:

  elasticsearch:
    container_name: elasticsearch
    image: 'docker.elastic.co/elasticsearch/elasticsearch:6.8.10'
    ports:
      - '9200:9200'
      - '9300:9300'
    mem_limit: 16GB
    environment:
      - discovery.type=single-node
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms8g -Xmx8g"      
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - 'esdata1:/usr/share/elasticsearch/data'
    restart: always

volumes:
  esdata1:
    driver: local

Example of Spring Stat document:

  • Shards = 1, replicas = 0
@Document(indexName = "stats_test", type = "stat", shards = 1, replicas = 0)
public class EsStat {

    @Id
    @Field(type = FieldType.Keyword)
    private String id;

    @Field(type = FieldType.Keyword)
    private String entityOrRelationshipId;

    @Field(type = FieldType.Keyword)
    private String articleId;

    @Field(type = FieldType.Keyword)
    private String status;

    @Field(type = FieldType.Keyword)
    private String type;

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
    @Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
    private ZonedDateTime date;

    @JsonProperty("type")
    @Field(type = FieldType.Keyword)
    private String dataSource;

    // getter and setters 
}

Stats Spring repository:

  • Indexing is done via Spring Data Elasticsearch repository:
public interface StatElasticsearchRepository extends ElasticsearchRepository<EsStat, String> {
}

Stats mapping:

{
  "stats": {
    "mappings": {
      "stat": {
        "properties": {
          "_class": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "articleId": {
            "type": "keyword"
          },
          "dataSource": {
            "type": "keyword"
          },
          "date": {
            "type": "date",
            "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
          },
          "entityOrRelationshipId": {
            "type": "keyword"
          },
          "id": {
            "type": "keyword"
          },
          "status": {
            "type": "keyword"
          },
          "type": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

What can I do to identify why the CPU usage is so high and what can I do to reduce it?

Any advice or recommendations would be much appreciated. I'm happy to add more configuration/output if required.

spring-data-elasticsearch elasticsearch
1 Answers

So let's assume that it's the refresh call that is done after every save() - It's quite easy to add a method to your repository that does a save without calling refresh:

First add a custom repository interface - I have it as a generic here:

CustomRepository.java:

public interface CustomRepository<T> {
    T saveNoRefresh(T entity);
}

The implementation would be:

CustomRepositoryImpl.java:

public class CustomRepositoryImpl<T> implements CustomRepository<T> {

    private final ElasticsearchOperations operations;

    public CustomRepositoryImpl(ElasticsearchOperations operations) {
        this.operations = operations;
    }

    @Override
    public T saveNoRefresh(T entity) {
        IndexQuery query = new IndexQueryBuilder().withObject(entity).build();
        operations.index(query, operations.getIndexCoordinatesFor(entity.getClass()));
        return entity;
    }
}

Your repository would change to:

public interface StatElasticsearchRepository 
  extends ElasticsearchRepository<EsStat, String>,
           CustomRepository<EsStat> {
}

The instead of the save(T) method call the saveNoRefresh(T) method of your repository.

Even better would be, if you can collect the data to batches of some size and then do a saveAll() call.

user_43935650
Related