Is there limitation for ES log ingestion in ES in Bulk API?

user_4153150 0
Viewed 138

I have task where I have to ingest 50 million logs. I have been using bulk API of ES and able to ingest only 60K in one go. I am using Curl to do that.

Here is all details:

curl  -u prvn:prvn -H 'Content-Type: application/json' -XPOST localhost:9200/_bulk --data-binary @IltUId9adHYNO3wnqixVzaBulk.json

IltUId9adHYNO3wnqixVzaBulk.json file contains logs in the format of Bulk API.

But here is an issue. When I am trying to ingest 70K logs in one go then "NOTHING" is happening. Curl command terminates without any issue and logs do not get ingested.

elasticsearch-5 elasticsearch
1 Answers

The maximum amount of data you can send in one go is governed by the http.max_content_length settings, which defaults to 100MB. So it looks like 60K lines is just below that limit and 70K lines just above it.

You can either increase the limit, which is usually not advised, or just send batches of 60K.

user_46045790
Related