dynamic URL parameters with logstash http-poller

user_890809 0
Viewed 4779

I would like to use logstash's http-poller input to consume a webservice returning an array of json documents to load data into elasticsearch. In order to do this, the webservice GET URL needs to be updated with URL query parameters between requests.

For example, the web service takes a date range specified as start and end parameters and limits responses to 1000 results. When first loading data, I would like logstash to continually call the web service with the start time getting set to the latest timestamp from the previous request until all data has been loaded.

To clarify, the first HTTP GET request should look like webserviceurl?start=2015-01-01T00:00:00. Assume the latest document in the response has the timestamp 2015-01-01T21:55:34. Then the next HTTP GET request should look like webserviceurl?start=2015-01-01T21:55:34

Is this possible with logstash and the http-poller input?

logstash
3 Answers

There is nothing like a variable declaration we can do in input .. a workaround rather is to define env variables with dates eg. on windows (powershell script) -

$env:startDate=(Get-Date).AddDays(-1).ToString('yyyy-MM-dd')
$env:endDate=(Get-Date).AddDays(0).ToString('yyyy-MM-dd')

Then we can use these variables as ${startDate} in the url . However once logstash is started the dates remain static. Guess we need to restart logstash script everyday for it to take the new value of date.

Another alternative is to write a proxy webservice, which will probably be in java or other languages , where the java class can be declared with variables and then it invokes the actual webservice and returns the response back to the logstash script.

This issue is pending in logstash since 2016... not sure why it cannot be addressed !

user_2362000

Dirty Workaround

My Use-Case

Call GitLab API to fetch JSON Array of some events by calling a URL with parameters after and before like:

curl -XGET "https://gitlab.com/api/v4/projects/<Project-ID-here>/events?target_type=issue&after=2020-04-14&before=2020-04-17" -H "PRIVATE-TOKEN: GitLab-Private-Token-Here"

Steps followed

First, use the File Input Plugin to read all files inside a given folder like:

input {
    # Read files: /home/thakur/gitlab_events/2020.04.14.json
    file {
        path => "/home/thakur/gitlab_events/*"
        codec => "json"
        mode => "read"
        #exit_after_read => true # it has a Bug
        close_older => 10
        sincedb_path => "/dev/null"
    }
}
output {
    stdout { codec => rubydebug }
    #elasticsearch {} # or wherever you want
}

After that, I created a Shell script and used Cron entry to create daily files of it:

#!/bin/bash

today=$(date -d '-0 day' '+%Y-%m-%d')
yesterday=$(date -d '-1 day' '+%Y-%m-%d')
day_before_yesterday=$(date -d '-2 day' '+%Y-%m-%d')
url="https://gitlab.com/api/v4/projects/<PROJECT-ID-HERE>/events?target_type=issue&after=$day_before_yesterday&before=$today"
curl -XGET $url -H "PRIVATE-TOKEN: YOUR-TOKEN-HERE" > /home/thakur/gitlab_events/$yesterday.json
user_35657560
Related