I have to bulk import data via LogStash as below :
Source: Restful Get APIs
Destination : ElasticSearch
My logstash config file looks as below:
input {
http_poller {
urls => {
test1 => {
method => get
url => "https://forun/questions?limit=100&offset=0"
headers => {
Accept => "application/json"
}
}
}
output {
elasticsearch{
hosts => ["localhost:9200"]
index => "testindex"
}
stdout {
codec => rubydebug
}
}
This fetches 200 records at a time.
But, I have more than 10000 records and need to apply pagination logic in input of http-poller plugin.
Please help how can I apply pagination logic here.