Cannot parametrize output Logstash index - AWS s3 input

Question

Good morning,

We are trying to collect logs from AWS s3 using Logstash. Here's the Logstash.conf file:

input {

  tcp {
    port => 5000
    codec => json
  }

  s3 {
    # Keys for user in prod env
    id => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

  s3 {
    # Keys for user in preprod env
    id => "preprod"
    access_key_id => "key2" 
    secret_access_key => "secretkey2"
    region => "eu-central-1"
    bucket => "bucket2"
    prefix => "prefix2/"
    include_object_properties => true
  }
  
}

filter {

  # Matching prod logs
  if [id] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  }
  else if [id] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }

}

output {
  opensearch {
    hosts       => "https://osb-platform"
      user => "username"
      password => "password"
    index       => "4s-%{[env]}-logs-%{+YYYY.MM.dd}"
    ecs_compatibility => disabled
    ssl_certificate_verification => false
  }
}

As you can see, I have two inputs from two different environments, and I want to put logs from different environments on different indexes. This is done through filtering on the id od the input, by adding a new field depending on the env.

Logs are correctly collected, but indexes are not distinguished: from OpenSearch I can see a single index called exactly 4s-%{[env]}-logs-2022.05.30, it's like it cannot resolve the env variable.

Can you please help me?

Thanks,

Luca

user_1123206 · Answer

The id option in any input is used to help identify the input while monitoring logstash, it is not used for filtering as it is not present in the document.

You need to use the tags or type options to do that.

For example, using type:

  s3 {
    # Keys for user in prod env
    type => "prod" 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

Then your filter would be:

filter {
  if [type] == "prod" {
    mutate {
      add_field => { "env" => "prod" }
    }
  } else if [type] == "preprod" {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }
}

And if you want to use tags, you would need something like this:

  s3 {
    # Keys for user in prod env
    tags => ["prod"] 
    access_key_id => "key1" 
    secret_access_key => "secretkey1"
    region => "eu-central-1"
    bucket => "buket1"
    prefix => "prefix1/"
    include_object_properties => true
  }

filter {
  if "prod" in [tags] {
    mutate {
      add_field => { "env" => "prod" }
    }
  } else if "preprod" in [tags] {
    mutate {
      add_field => { "env" => "preprod" }
    }
  }
}

Cannot parametrize output Logstash index - AWS s3 input

1 Answers