Bulk import data via LogStash

Viewed 692

I have to bulk import data via LogStash as below :

Source: Restful Get APIs

Destination : ElasticSearch

My logstash config file looks as below:

input {
  http_poller {
    urls => {
     test1 => {
       method => get
       url => "https://forun/questions?limit=100&offset=0"
       headers => {
         Accept => "application/json"
    }
  }
}

output {
elasticsearch{
 hosts => ["localhost:9200"]
 index => "testindex"
}
  stdout {
    codec => rubydebug
  }
}

This fetches 200 records at a time.

But, I have more than 10000 records and need to apply pagination logic in input of http-poller plugin.

Please help how can I apply pagination logic here.

1 Answers

I'm not sure if this is the right architecture. Why have you decided to pull data into Logstash (using the http_poller input plugin) instead of pushing data to Logstash?

If it's a single bulk upload; you could just script something to send events directly to Logstash over http (using http input plugin). Alternatively if it's not a single bulk upload and there's a lot of data that grows over time, you might consider sending events to something like Kafka/Redis/RabbitMQ, which can get streamed to Logstash using associated input plugins.

Related