Source to destination Key Field mapping in Elastic Search

Ankita Mehta 0
Viewed 39

I have a elastic search index with source data coming in the following way:

        "_source": {
            "email": "smithamber@example.com",
            "time": "2022-09-08T13:52:50.347861",
            "message": "Pattern thank talk mention. Manage nearly tell beat. Difficult husband feel talk radio however.",
            "sIp": "192.168.11.156",
            "dIp": "80.254.211.60",
            "ts": "2022-09-08T13:52:50"
        }

Now I want a way to treat dynamically map @timestamp [destination key] field of ES doc to be time [source key]. For this i am using:

 "runtime_mappings": {
        "@timestamp": {
            "type": "date",
            "format": "yyyyMMdd'T'HHmmss.SSSZ",
            "script": {
                "source": "if (doc[\"time\"].size() == 0) {return} else {return doc[\"time\"].value;}",
                "lang": "painless"
             }
            }
        }

However, this does not work. Is there a better way to map source key field to destination key field in elastic search. I am open to static mapping as well if we set once before creating the index for one kind of source data.

I am looking for correct syntax for mapping my field.

Edited: When I add the query -

{     "query": {
    "range": {
      "@timestamp": {
        "gte": "now-5d",
        "lte": "now"
      }
    }
  }
}

I see no hits.

{
"took": 20,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": {
        "value": 0,
        "relation": "eq"
    },
    "max_score": null,
    "hits": []
}
}

However, same query on field time gets me all filtered docs.

{
"took": 27,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": {
        "value": 10000,
        "relation": "gte"
    },
    "max_score": 1.0,
    "hits": [
        {
            "_index": "topic-indexer-xxx",
            "_id": "c28sIYMB0xJUJru8c47O",
            "_score": 1.0,
            "_source": {
                "email": "albertthompson@example.com",
                "time": "2022-09-07T15:25:33.672016",
                "message": "Candidate future staff ever former run. Like quality personal specific trouble cell money move. Available majority memory model thing TV wrong. Summer anyone light key.",
                "sIp": "192.168.103.75",
                "dIp": "191.27.68.163"
            }
        },
  ....
  }

For mapping I have also tried dynamic templates; but still no results on query for @timestamp field:

{
"dynamic_templates": [
  {
    "@timestamp": {
      "match":   "time",
      "mapping": {
        "type":       "date",
        "format": "strict_date_optional_time",
        "copy_to":    "@timestamp"
      }
    }
  }
]
}

With @paulo's response, I just did a little fine tuning to resolve the issue; The below mapping (as set) works and then I can run range queries on the @timestamp field:

{
  "runtime": {
    "@timestamp": {
      "type": "date",
      "script": {
        "source": "if (doc['time'].size() != 0){ emit(doc['time'].value.toEpochMilli());}",
        "lang": "painless"
      }
    }
  },
  "properties": {
      "@timestamp": {
          "type": "date"
      }
  }
}
mapping elasticsearch
1 Answers

Tldr;

I feel you go mixed up in your painless script. Please find below an example you should be able to reproduce on your side.

Time is already a date on my side. Elasticsearch was able to detect it automatically.

On another note, using runtime fields while very flexible, may lead to performance issue on the long run.

Maybe you should be looking into ingest pipeline.

Solution

POST /73684302/_doc
{
  "email": "smithamber@example.com",
  "time": "2022-09-08T13:52:50.347861",
  "message": "Pattern thank talk mention. Manage nearly tell beat. Difficult husband feel talk radio however.",
  "sIp": "192.168.11.156",
  "dIp": "80.254.211.60",
  "ts": "2022-09-08T13:52:50"
}

POST /73684302/_doc
{
  "email": "smithamber@example.com",
  "message": "Pattern thank talk mention. Manage nearly tell beat. Difficult husband feel talk radio however.",
  "sIp": "192.168.11.156",
  "dIp": "80.254.211.60",
  "ts": "2022-09-08T13:52:50"
}

GET /73684302/_search
{
  "runtime_mappings": {
    "@timestamp": {
      "type": "date",
      "script": {
        "source": """
        if (doc["time"].size() != 0){
          emit(doc["time"].value.toEpochMilli());
        }
        """,
        "lang": "painless"
      }
    }
  },
  "_source": false, 
  "fields": ["@timestamp"]
  
}
userxL1j0
Related