Store Map with Collection as Value in Elasticsearch

userVNwf 0
Viewed 17

I have a problem storing private Map<String, Set<String>> in elastic.

For example, I have a variable private Map<String, Set<String>> updatedFields with the following content:

  "updatedFields": {
    "key1": [
      "val1",
      "val2",
      "val3"
    ],
    "key2": [
      "val1",
      "val2",
      "val3"
    ]
  }

With the template mapping "updatedFields": {"type": "object"} elastic will create automappings for every key stored. This will lead to the following error if more than 1000 unique keys are stored:

failure in bulk execution:
[0]: index [md_activity_log_stg_ii], type [_doc], id [969d12df-d2c4-446f-975a-d95e76d6e720], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded while adding new fields [1]]];]

By changing the setting to "updatedFields": {"type": "object", "enabled": false} the object is stored but without any content of the array itmes Example of a document in Elastic after storing:

"updatedFields" : {
      "34b49fbb-7ca8-4dc2-8c1f-21f88324b393" : [ ]
 },

Anyone knows you to configure the field so that map is correctly stored without exceeding the field limit?

java elasticsearch
1 Answers

I assume you are retrieving JSON and processing it in Java. Make sure you defined or only capture the fields or nested fields you wanted by defining POJOs. Without this, ElasticSearch would dynamically map any keys inserted resulting to mapping explosion.

Here are the list of references in ElasticSearch that might help you fix your problem:

Adding explicit mapping for objects https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html

Adding a limit to avoid mapping explosion https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings

Adding explicit mapping in general https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html

Clark Ngo0
Related