Syncing the data from mongoDB to elasticsearch using Logstash

user_7509966 0
Viewed 4781

I am trying to sync the mongoDB database to the elasticsearch. I am using logstash-input-mongoDb and logstash-output-elasticsearch plugins.

The issue is mongoDb plugin is not able to extract all the information from the inserted document in mongodb, thus I am seeing only few fields being inserted to the elasticsearch. And I also get the entire query as the log in elasticsearch index. I tried to manipulate the filters in the config file for the logstash and change the input to the elasticsearch but could not make it work.

Any help or suggestion would be great.



Edit:
Mongo schema:

A:{
  B: 'sometext',
  C: {G: 'someText', H:'some text'}
},
D:[
 {E:'sometext',F:'sometext'},
 {E:'sometext',F:'sometext'},
 {E:'sometext',F:'sometext'}
]

plugin:

    input {
    mongodb {
        uri => 'mongodb://localhost:27017/testDB'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'testCOllection'
        batch_size => 1000
    }
}
output {
        stdout {
                codec => rubydebug
        }
        elasticsearch {
                action => "index"
                index => "testdb_testColl"
                hosts => ["localhost:9200"]
        }
}

output to elastic:

{
    //some metadata
    A_B: 'sometext',
    A_C_G: 'someText',
    A_C_H: 'some text',
    log_entry: 'contains complete document inserted to mongoDB'
}

We are not getting property D of mongo collection in the elastic. Hope this explains the problem more elaborately.

logstash elasticsearch mongodb plugins
1 Answers

because your configuration looked good to me, I checked the issues of the phutchins/logstash-input-mongodb repo, and I found this one: "array not stored to elasticsearch", which pretty much described your problem. It is still an open issue, but you might want to try out the workaround suggested by ivancruzbht. Such workaround uses the ruby Logstash filter to parse the log_entry field, which you also confirmed has all the fields - including D.

user_30293370
Related