I am learning the ELK stack. At this time, I have Elasticsearch and Kibana setup on an AWS instance. I am successfully writing docs to an Elasticsearch index on that index via a Java console app I've written. I can see those logs in Kibana. Now, I want to see those logs input into Logstash on a different machine.
In an attempt to do this, I've installed Logstash. I added the following configuration:
logstash.conf
input {
elasticsearch {
hosts => "https://<aws-instance>.us-east-1.aws.found.io:9243"
user => "<myUsername>"
password => "<myPassword>"
index => "<myIndexName>"
query => '{ "query": { "match": { "statuscode": 200 } }, "sort": [ "_doc" ] }'
}
}
output {
stdout { }
}
When I start Logstash from a console window, I'm expecting to see logs pushed to my Elasticsearch index appear. Instead, I see:
Sending Logstash logs to C:/Temporary/logs which is now configured via log4j2.properties
[...][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[...][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[...][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[...][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x13bbcec sleep>"}
[...][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[...][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[...][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#<Thread:0x13bbcec run>"}
I don't understand this. Why would the "pipeline" terminate? At this point, I don't understand if the issue is with Logstash or connecting to my Elasticsearch index. In either case, I have no idea. I also tried running with the trace log level. Unfortunately, I didn't learn anything from those additional logs. There are no ERROR messages. After the initialization of the Elasticsearch input plugin, I simply see:
[...][DEBUG][logstash.inputs.elasticsearch] Closing {:plugin=>"LogStash::Inputs::Elasticsearch"}
I don't know why it would close though.