Using OpenSearch Python bulk api to insert data to multiple indices

user_5807238 0
Viewed 1431

This document shows how bulk data with multiple index can be inserted using POST request in curl: https://opensearch.org/docs/latest/opensearch/index-data/

If I have data in this format,

[
{ "index": { "_index": "index-2022-06-08", "_id": "<id>" } }
{ "A JSON": "document" }
{ "index": { "_index": "index-2022-06-09", "_id": "<id>" } }
{ "A JSON": "document" }
{ "index": { "_index": "index-2022-06-10", "_id": "<id>" } }
{ "A JSON": "document" }
]

Bulk request should take the index name from "_index": "index-2022-06-08"

I was trying to use OpenSearch-py library to do the same but I can't find any example snippet does that. I am using this format to send request from AWS Lambda.

client = OpenSearch(
            hosts = [{'host': host, 'port': 443}],
            http_auth = awsauth,
            use_ssl = True,
            verify_certs = True,
            connection_class = RequestsHttpConnection
            )
        
        resp = helpers.bulk(client, logs, index= index_name, max_retries = 3)

Here, I've to mention index_name as a parameter in bulk request so it's not taking index_name from data itself. If I don't mention index_name in parameter, I get error 4xx index_name missing.

I was also looking into bulk api source code: https://github.com/opensearch-project/opensearch-py/blob/main/opensearchpy/helpers/actions.py#L373

It doesn't look like index_name is a mandatory parameter.

Can anyone help me with what am I missing?

elasticsearch opensearch rest api
1 Answers

I came across the same issue and found the solution in the elasticsearch.py bulk-helpers documentation. When the documents are provided in the _source-structure that the search-endpoint returns it works.

Call of the bulk-method:

            resp = helpers.bulk(
                self.opensearch,
                actions,
                max_retries=3,
            )

Where actions is a list of dictionaries like this:

[{
    '_op_type': 'update',
    '_index': 'index-name',
    '_id': 42,
    '_source': {
        "title": "Hello World!",
        "body": "..."
    }
}]

_op_type can be used as an additional field to define the action(index, update, delete,...) that should be invoked for the document.

Hope this helps anyone coming across the same issue!

user_59300470
Related