I'm using ELK for reporting purpose. Logstash JDBC plugin used to feed elastic search from Oracle query. I'm having the index name with daily date as incremental postfix. And I'm using document ID as primary key from DB. But same record in DB will go through changes over the time. So i'm having hourly schedule on logstash input and DB query also to take records which updated in last 1 hour. Most of the records are being updated over a week time. Because of this reason the same record reflected in multiple indexes. Is there a way to make the document ID unique across all indexes?
Logstash config
input{
jdbc{
jdbc_driver_library=>"ojdbc8.jar"
jdbc_driver_class=>"Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string=>"db connection string"
jdbc_user=>"user"
jdbc_password=>"pass"
statement_filepath=>"incident.sql"
schedule=>"0 * * * *"
id=>"incident_details"
type=>"incident_details"
tracking_column_type=>"numeric"
tracking_column=>"incidnet_id"
}
}
if[type]="incident_details"{
elasticsearch{
index=>"devops-servicenow-%{[type]}-%{+YYYY.MM.dd}"
hosts=>["elk1.mydomain.com:9200","elk2.mydomain.com:9200"]
document_id=>"%{incidnet_id}"
doc_as_upsert=>true
action=>"update"
}
}
SQL
SELECT
incident_id,
created_time,
modified_time,
incident_status
FROM incident WHERE modified_time BETWEEN (SYSDATE-1/24) AND SYSDATE;