Elasticsearch 7.13 populating(enrich) index with matching field values from another index

user_15799164 0
Viewed 260

I will write my question as is;

There are two indices under your cluster; earthquakes and magnitude_type

lets say earthquakes index has bunch of fields and a "MagnitudeType" magnitude_type index has "MagnitudeType" and "MagnitudeTypeDesc"

you want to create a new index named my_index; my_index should have all the documents from earthquakes, we will add "MagnitudeTypeDesc" field to my_index and then populate it according to magnitude_type index where "MagnitudeType" field matches.

How can one do this?

Thanks

UPDATE!!!!

I used Val's suggestion as below,

PUT /_enrich/policy/second-index-policy
{
"match": {
  "indices": "magnitude_type",
   "match_field": "MagnitudeType",
   "enrich_fields": ["MagnitudeTypeDesc"]
 }
}
PUT /_enrich/policy/second-index-policy/_execute

PUT /_ingest/pipeline/second_index_lookup
{
 "processors" : [
   {
     "enrich" : {
       "description": "Update data from the second index",
       "policy_name": "second-index-policy",
       "field" : "MagnitudeType",
       "target_field": "MagnitudeType"
  }
}
]
}

POST _reindex
{
 "source": {
   "index": "earthquakes"
},
 "dest": {
  "index": "new_index",
  "pipeline": "second_index_lookup"
}
}

when i do this, id writes both fields to MagnitudeType. my _source looks like this what am i doing wrong?

"_source" : { "MagnitudeType" : { "MagnitudeType" : "MW", "MagnitudeTypeDesc" : "cok fena zelzele" }, "name" : "can" }

elasticsearch
1 Answers

You did everything right by mimicking what this thread advocates for, i.e. using an enrich processor to join data from the magnitude_type index into the new_index.

Now you simply need to flatten down the matched magnitude object using another processor called rename and also remove the temporary result object from the enrich processor, like this:

PUT /_ingest/pipeline/second_index_lookup
{
 "processors" : [
   {
     "enrich" : {
       "description": "Update data from the second index",
       "policy_name": "second-index-policy",
       "field" : "MagnitudeType",
       "target_field": "tmp"
     }
   },
   {
     "rename": {
       "field": "tmp.MagnitudeType",
       "target_field": "MagnitudeType"
     }
   },
   {
     "rename": {
       "field": "tmp. MagnitudeTypeDesc",
       "target_field": "MagnitudeTypeDesc"
     }
   },
   {
     "remove": {
       "field": "tmp"
     }
   }
 ]
}

After that, your source should look like this:

{ 
   "MagnitudeType" : "MW", 
   "MagnitudeTypeDesc" : "cok fena zelzele",
   "name" : "can" 
}
user_46045790
Related