post request with \n-delimited JSON in python

user_863713 0
Viewed 3607

I'm trying to use the bulk API from Elasticsearch and I see that this can be done using the following request which is special because what is given as a "data" is not a proper JSON, but a JSON that uses \n as delimiters.

curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d '
{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
'

My question is how can I perform such request within python? The authors of ElasticSearch suggest to not pretty print the JSON but I'm not sure what it means (see https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)

I know that this is a valid python request

import requests
import json

data = json.dumps({"field":"value"})

r = requests.post("localhost:9200/_bulk?pretty", data=data)

But what do I do if the JSON is \n-delimited?

python-requests elasticsearch python json
3 Answers

As a follow-up to Daniel's answer above, I had to add an additional '\n' to the end of the data_to_post, and add a {Content-Type: application/x-ndjson} header to get it work in Elasticsearch 6.3. 

data_to_post = '\n'.join(json.dumps(d) for d in data) + "\n"
headers = {"Content-Type": "application/x-ndjson"}
r = requests.post("http://localhost:9200/_bulk?pretty", data=data_to_post, headers=headers)

Otherwise, I will receive the error: "The bulk request must be terminated by a newline [\\n]"

user_106276630

You can use python ndjson library to do it. https://pypi.org/project/ndjson/

It contains JSONEncoder and JSONDecoder classes for easy use with other libraries, such as requests:

import ndjson
import requests

response = requests.get('https://example.com/api/data')
items = response.json(cls=ndjson.Decoder)
user_66688950
Related