logstash + elasticsearch : reloads the same data

Viewed 4556

Managed to get logstash (1.3.1) to send data to elasticsearch (0.9.5).

My logstash conf file setup is

input {
  file {
    path => ["D:/apache-tomcat-7.0.5/logs/*.*"]
   }

}
 output {
  stdout { } 
    elasticsearch_http {
    host => "localhost"
    port => 9200
   }
 }

The data is stored in ES under index logstash-2013.12.xx

However, if i restart logstash, lets say next day - the same data is reloaded into a new index. Even if i restart again, the document count doubles in the index.

Seems like logstash re-reading the data and ES is also duplicating the documents.

Is there a way to not reload in logstash or not duplicate in ES or do BOTH.

1 Answers
Related