I set filebeat to read logs from rotating logs (rotated when 5 mg is reached) and below is my config :
- type: log
fields:
source: 'filebeat2'
logID: logbackup
fields_under_root: true
enabled: true
paths:
- /home/logbackup/a.log
-/home/logbackup/backup/a.log-*
output.logstash:
# The Logstash hosts
hosts: ["ip:5044"]
worker: 4
bulk_max_size: 4096
queue:
mem:
events: 16384
and logstash.yml :
pipeline.workers: 4
pipeline.batch.size: 4096
and close-inactive is default(5min).we have 100 transaction per second .I stop filebeat manually for specific time(for crash test) and when start it manually (with 2 million docs stored in second directory path )and some logs been duplicated. what is the solution ,is it possible solution to increase close-inactive time ?