ElasticSearch Spring - disable date_detection only for a set of fields and not the entire index using @Mapping annotation

user_18438839 0
Viewed 127

I'm trying to disable date_detection on a set of fields in an index. Below is the mapping

{
  "my-index" : {
    "mappings" : {
      "properties" : {
        "_class" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "customFields" : {
          "properties" : {
            "firstName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "lastName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "address" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "dateOfBirth" : {
              "type" : "date"
            }
          }
        },
        "key" : {
          "type" : "long"
        },
        "updatedDate" : {
          "type" : "date",
          "format" : "basic_date_time"
        }
      }
    }
  }
}

I want the dateOfBirth field to be of type text, not date.

So I did the following:

I created a mappings.json file (see below) and used the @Mapping(mappingPath = "mappings.json") annotation

{
  "date_detection": false
}

Now, this does disable date_detection but it also forces the updatedDate to be of type text and this is causing some errors.

These are the updatedDate and customFields variables in my index class:

@Field(type = FieldType.Date, format = DateFormat.basic_date_time)
Instant updatedDate;

Map<String, Object> customFields;

Is there a way to disable date_detection for the fields inside customFields so that only the dateOfBirth field's type is changed to text and not updatedDate?

elastic-stack spring-data-elasticsearch elasticsearch
2 Answers

Reading the documentation at https://www.elastic.co/guide/en/elasticsearch/reference/8.2/dynamic-field-mapping.html#_disabling_date_detection date_detection is per index and not configurable for individual fields.

If you use a mappings file, Spring Data Elasticsearch does not create a mapping from the entity but only uses the file, so your @Field annotation is ignored.

Keep the @Field annotation on the updatedDate and add

@Mapping(dateDetection = Mapping.Detection.FALSE)

to your entity class (the one annotated with @Document). Remove the reference to the mapping.json.

user_43935650

The answer provided by P.J. Meisch works great. However older versions of Spring Data Elasticsearch don't contain the dateDetection attribute of the @Mapping annotation. If you can't upgrade Spring Data Elasticsearch, you can annotate fields with:

@Field(type = FieldType.Text, store = true)

This will prevent Elasticsearch from inferring the data type of a text field based on it's value and mapping it as a date if it contains data in a common date format. This is useful if you have a String field with records that may sometimes contain data that is in a valid date format and other times not. In those situations, the automatic mapping of fields based on value is problematic because if the first value indexed by Elastic happens to be a valid date format, it will map the field as a date. Subsequent records that are indexed that do not contain valid date data will be rejected since they can't be parsed as a date.

According to the documentation, the store attribute is used to "Flag whether the original field value should be store in Elasticsearch, default value is false". So while the name of the attribute may lead one to believe it's used to mark a field as transient or something similar, it appears it's actually used to tell Elastic to just index the field value verbatim rather than evaluating the value to infer it's type.

user_19707460
Related