accessing the previous value in a field logstash

Question

my document in elasticsearch looks like this:

"_index" : "micro_youtube",
"_type" : "video",
"_id" : "-i4trOrPnDE",
"_score" : 1.0,
"_source" : {
  "id" : "-i4trOrPnDE",
  "title" : "HubSpot’s Data Security Pillars to Safeguard Your Data",
  "post_time" : "2022-05-11T12:51:22Z",
  "body" : "Looking for a marketing automation solution that multiplies your marketing efforts and keeps your data safe at the same time? If yes, HubSpot is your best bet. Learn how HubSpot secures your data with its out-of-the-box features.",
  "tag" : [
    "bootle",
    "mouse"
  ],
  "view_href" : "https://www.youtube.com/watch?v=-i4trOrPnDE",
  "channel_id" : "UCpf-e6rXvNOS_VHSgZT3Yqg",
  "user_id" : "pf-e6rXvNOS_VHSgZT3Yqg",
  "comment" : "",
  "subtitle" : "",
  "thumbnail" : "https://i.ytimg.com/vi/-i4trOrPnDE/mqdefault.jpg",
  "privacy_status" : "public",
  "channelType" : "Subscribed",
  "indexedDate" : "2022-06-29T04:59:56.104Z",
  "language" : "en",
  "Language" : "en",
  "objectName" : "video",
  "indexName" : "micro_youtube",
  "uniqueField" : "micro_youtube_video_-i4trOrPnDE",
  "value_to_append" : [
    "bottle",
    "mouse"
  ],
  "tags" : [
    "_elasticsearch_lookup_failure"
  ]

I am using Logstash to append a new value at the end of the "tag" field.

My Logstash.conf file looks like this:

input {
    kafka {
        bootstrap_servers => "http://kafka:9092"
        topics => "elasticIndexUpdateTopic"
        codec => json {}
    }
}

filter {
    elasticsearch {
        hosts => ["http://elasticsearch:8045"]
        index => "micro_youtube"
        query_template => "query-template.json"
        fields => {
            "[tag]" => "tag"
        }
    }

    mutate {
        remove_field => [ "@timestamp", "@version" ]
        merge => ["tag", "value_to_append"]
    }
}

I am getting the value of the field "value_to_append" from the backend. Its value can be anything like ["ABC", "XYZ"], etc.

The expected result with the above setup if "value_to_append" is ["ABC", "XYZ"] should be:

[ "bottle", "mouse", "ABC", "XYZ" ]

But, in the current scenario, it is simply replacing the previous value of the "tag" field with the value of the field "value_to_append" and giving the "_elasticsearch_lookup_failure" error.

Is there a way by which I can achieve this thing?

accessing the previous value in a field logstash

0 Answers