I am trying to run logstash:latest container, but pipeline is throwing an error when I run it.
When I run it directly with -f "/path/to/pipeline", everything works. Once I run it from pipelines.yml, it says the pipeline has a bad formatting on the first line, first column.
14:39:28.728 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
NOTE: Before I even had a comment on the first line and it said the error is after comment, so file access and path are definitely okay.
If anyone has an idea to how or why I am experiencing this behavior, I am open to suggestions. At the bottom I am including my configurations.
pipelines.yml
- pipeline.id: log-pipeline
path.config: "/etc/logstash/pipelines/log_elastic_write.conf"
pipeline.workers: 1
log_elastic_write.conf
input {
kafka {
bootstrap_servers => "kafka:9092"
topics => ["logs"]
group_id => "logs_write"
auto_offset_reset => "earliest"
}
}
filter {
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSSSSS" ]
timezone => "UTC"
}
}
filter {
json {
source => "message"
}
}
filter {
json {
source => "message"
target => "raw_message"
}
mutate {
rename => {"@timestamp" => "timestamp_message_received"}
}
mutate {
remove_field => [ "message" ]
rename => {"[raw_message][timestamp]" => "timestamp_message_sent"}
}
}
filter {
date {
match => ["timestamp_message_sent","yyyy-MM-dd HH:mm:ss.SSS"]
target => "timestamp_message_sent"
locale => "en"
}
}
filter {
mutate { remove_field => [ "raw_message" ] }
mutate { remove_field => [ "timestamp" ] }
}
output {
elasticsearch {
hosts => "ipelastic:82"
index => "logging"
}
}
entrypoint.sh
#!/usr/bin/env bash
echo 'Inspecting file structure'
find /etc/logstash/ -maxdepth 4 | grep -v git | grep -v idea
echo 'Setting up runtime and pipeline configuration files(entrypoint.sh):'
echo "---------- 'logstash.yml' ----------"
envsubst < "${LOGSTASH_DIR_TARGET}/${LOGSTASH_CONFIG_SOURCE}" > "${LOGSTASH_DIR_TARGET}/${LOGSTASH_CONFIG_TARGET}"
rm "${LOGSTASH_DIR_TARGET}/${LOGSTASH_CONFIG_SOURCE}"
echo "---------- 'pipelines.yml' ----------"
envsubst < "${LOGSTASH_DIR_TARGET}/${PIPELINES_CONFIG_SOURCE}" > "${LOGSTASH_DIR_TARGET}/${PIPELINES_CONFIG_TARGET}"
rm "${LOGSTASH_DIR_TARGET}/${PIPELINES_CONFIG_SOURCE}"
less "${LOGSTASH_DIR_TARGET}/${PIPELINES_CONFIG_TARGET}"
echo "---------- pipeline: elasticsearch logs ----------"
envsubst < "${PIPELINE_DIR_TARGET}/${LOG_WRITE_PIPELINE_SOURCE}" > "${PIPELINE_DIR_TARGET}/${LOG_WRITE_PIPELINE_TARGET}"
rm "${PIPELINE_DIR_TARGET}/${LOG_WRITE_PIPELINE_SOURCE}"
less "${PIPELINE_DIR_TARGET}/${LOG_WRITE_PIPELINE_TARGET}"
chmod a+rx "${LOGSTASH_DIR_TARGET}/${LOGSTASH_CONFIG_TARGET}"
chmod a+rx "${LOGSTASH_DIR_TARGET}/${PIPELINES_CONFIG_TARGET}"
chmod a+rx "${PIPELINE_DIR_TARGET}/${LOG_WRITE_PIPELINE_TARGET}"
logstash