Logstash Docker Compose simplest stdin input and standard output example

Viewed 1571

I am trying to create a simplest example of Logstash on Docker Compose which will take input from stdin and give output to standard out. It's not working as expected. The following are the config files

docker-compose.yml

version: '3.7'
services:
  logstash-sandbox:
    image: docker.elastic.co/logstash/logstash:7.8.0
    container_name: logstash-sandbox
    user: root
    volumes:
      - ./logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./conf:/var/logstash/configuration
    ports:
      - 9600:9600
      - 5044:5044
    networks:
      - host

networks:
  host:

logstash.yml

http.host: 0.0.0.0
xpack.monitoring.enabled: false
xpack.management.enabled: false

config.reload.automatic: true
config.reload.interval: 6s

log.level: info
log.format: json

path.config: /var/logstash/configuration/*.conf

filter in conf folder

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
      stdout {}
}

Logs

$ docker-compose up  
Creating network "agents_host" with the default driver
WARNING: Found orphan containers (filebeat-sandbox, kibana-sandbox, es01-sandbox) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating logstash-sandbox ... done
Attaching to logstash-sandbox
logstash-sandbox    | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-sandbox    | WARNING: An illegal reflective access operation has occurred
logstash-sandbox    | WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
logstash-sandbox    | WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
logstash-sandbox    | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash-sandbox    | WARNING: All illegal access operations will be denied in a future release
logstash-sandbox    | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.setting.writabledirectory","timeMillis":1594296137893,"thread":"main","logEvent":{"message":"Creating directory","setting":"path.queue","path":"/usr/share/logstash/data/queue"}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.setting.writabledirectory","timeMillis":1594296137959,"thread":"main","logEvent":{"message":"Creating directory","setting":"path.dead_letter_queue","path":"/usr/share/logstash/data/dead_letter_queue"}}
logstash-sandbox    | {"level":"WARN","loggerName":"logstash.config.source.multilocal","timeMillis":1594296138392,"thread":"LogStash::Runner","logEvent":{"message":"Ignoring the 'pipelines.yml' file because modules or command line options are specified"}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.runner","timeMillis":1594296138398,"thread":"LogStash::Runner","logEvent":{"message":"Starting Logstash","logstash.version":"7.8.0","jruby.version":"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Server VM 11.0.7+10-LTS on 11.0.7+10-LTS +indy +jit [linux-x86_64]"}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.agent","timeMillis":1594296138432,"thread":"LogStash::Runner","logEvent":{"message":"No persistent UUID file found. Generating new UUID","uuid":"a7b0afbc-b2ab-4019-8a23-a771ae0513d5","path":"/usr/share/logstash/data/uuid"}}
logstash-sandbox    | {"level":"INFO","loggerName":"org.reflections.Reflections","timeMillis":1594296140309,"thread":"Converge PipelineAction::Create<main>","logEvent":{"message":"Reflections took 44 ms to scan 1 urls, producing 21 keys and 41 values "}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.javapipeline","timeMillis":1594296141109,"thread":"[main]-pipeline-manager","logEvent":{"message":"Starting pipeline","pipeline_id":"main","pipeline.workers":8,"pipeline.batch.size":125,"pipeline.batch.delay":50,"pipeline.max_inflight":1000,"pipeline.sources":["/var/logstash/configuration/kong-filter.conf"],"thread":"#<Thread:0x8104977 run>"}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.javapipeline","timeMillis":1594296142209,"thread":"[main]-pipeline-manager","logEvent":{"message":"Pipeline started","pipeline.id":"main"}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.agent","timeMillis":1594296142311,"thread":"Agent thread","logEvent":{"message":"Pipelines running","count":1,"running_pipelines":[{"metaClass":{"metaClass":{"metaClass":{"running_pipelines":"[:main]","non_running_pipelines":[]}}}}]}}
logstash-sandbox    | {"level":"INFO","loggerName":"logstash.agent","timeMillis":1594296142656,"thread":"Api Webserver","logEvent":{"message":"Successfully started Logstash API endpoint","port":9600}}




127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

I do not get any output as expected.

0 Answers
Related