why I have log duplication when filebeat read rotating log files and stop filebeat for specific time manually and default close inactive reached

Viewed 15

I set filebeat to read logs from rotating logs (rotated when 5 mg is reached) and below is my config :

- type: log
  fields:
  source: 'filebeat2'
  logID: logbackup
  fields_under_root: true
  enabled: true
  paths:
  - /home/logbackup/a.log
  -/home/logbackup/backup/a.log-*
  output.logstash:
  # The Logstash hosts
  hosts: ["ip:5044"]
  worker: 4
  bulk_max_size: 4096
   queue: 
     mem:
      events: 16384

and logstash.yml :

pipeline.workers: 4
pipeline.batch.size: 4096

and close-inactive is default(5min).we have 100 transaction per second .I stop filebeat manually for specific time(for crash test) and when start it manually (with 2 million docs stored in second directory path )and some logs been duplicated. what is the solution ,is it possible solution to increase close-inactive time ?

0 Answers
Related