logstash filter how to create two (or above ) outputs for the one input

user_368440 0
Viewed 980

I am getting via http poller one json 

  {
 "id":12345
 "name":"",
 "lastname":"",
 "age":12,
 "address":{"city":"XXXX" , "street":"ZZZZ" }
 }

and I would like this to generate two document in my output :

person : 

 {
"id":12345
"name":"",
"lastname":"",
"age":12
  }

address : 

 {
"city":"XXXX" , 
"street":"ZZZZ" 
  }

meaning I got one event in the input

in the input phase getting one input : 

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
}

in the filter phase I would like to :

  1. create person event (tag it as P)
  2. create address event (tag it as A)

in the output phase I would like to :

  1. send P to P type in ES
  2. send A to A type in ES
logstash-configuration logstash elasticsearch
1 Answers

You can achieve that with the clone filter.

First, you need to install the plugin which is not bundled by default:

bin/logstash-plugin install logstash-filter-clone

Then you can modify your Logstash config like this:

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
    type => "A"
  }
}
filter {
    clone {
        clones => [ "P" ]
        add_tag => [ "P" ]
    }
    if [type] == "P" {
        mutate {
            remove_field => [ "address" ]
        }
    } else {
        mutate {
            add_tag => [ "A" ]
            remove_field => [ "id", "name", "lastname", "age" ]
        }
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        document_type => "%{type}"
    }
}
user_46045790
Related