Can't assign elastic search index for data stream

user_5323305 0
Viewed 920

I am trying to create an index in elasticsearch using the index => option for elasticsearch logstash output running on docker:

output {
    elasticsearch {
        cloud_id => "..."
        data_stream => "true"
        ssl => "true"
        api_key => "..."
        document_id => "%{_log_id}"

        index => "%{target_index}"
    }
}

If I comment the index line, the pipeline works and data is sent to the default index. However, with the index defined (with or without it being a constant string) the following error is given on launch before ingesting any data

elasticsearch - Invalid data stream configuration, following parameters are not supported: {"index"=>"%{target_index}"}

Where target_index is an entry in the JSON body parsed in filter.

And breaks with Could not execute action: PipelineAction::Create<firmware_pipeline> indicating that this is before the pipeline is actually triggered.

Not sure if I'm just reading the docs wrong but this seems to be what others are doing as well.

Logstash version: 7.13.2

logstash elasticsearch
1 Answers

When you use a data stream, events are automatically routed to indexes based on values in the [data_stream] field. You cannot have automatic routing at the same time as explicit routing with the index => "%{target_index}" option. That is what the following is telling you:

following parameters are not supported: {"index"=>"%{target_index}"}

Remove the index option if you want to use a data stream. If you want explicit routing, remove the data_stream option.

If you need data to go to both destinations, use a second output.

user_117929770
Related