Cosmos Db - insert if not exists

Viewed 2889

Dear Cosmos fellow members,

I am creating Azure Function (Python) with CosmosTrigger that:

  • reads data from source_collection
  • writes data to target_collection (only if document does not exist)

To achieve "insert if not exists" functionality I configured target_collection with unique key "name".

Example:

  1. write to target_collection {"name": "John", "id": "1"} => OK
  2. write to target_collection [{"name": "John", "id": "2"}, {"name": "Mary", "id": "3"}] => System.Private.CoreLib: Exception while executing function: Functions.CosmosTrigger. Microsoft.Azure.DocumentDB.Core: Entity with the specified id already exists in the system.

Unique key "name": "John" already exists in target_collection. So outdocs: func.Out[func.Document], outdocs.set() returns conflict 409 and "Mary" never gets written to target_collection.

Important: when I upload json with array of items from Data Explorer to target_collection, all conflicts return an error, but all items get processed. When I try to write it from Azure function, first conflict raises an error and remaining items never make it to target_collection.

My questions:

  1. Is there a way to ignore raised errors and process whole array of items with func.Out[func.Document].set() ?
  2. Alternatively could you please advise me what would be the best way to implement bulk "insert if not exist" behaviour in Cosmos?
def main(docs: func.DocumentList, outdocs: func.Out[func.Document]) -> str:
  compl_docs = func.DocumentList()

    compl_docs_dict = {
      "name": "John",
      "id": "2"
    }

    compl_docs.append(func.Document.from_dict(compl_docs_dict))

    compl_docs_dict = {
      "name": "Mary",
      "id": "3"
    }

    compl_docs.append(func.Document.from_dict(compl_docs_dict))

  outdocs.set(compl_docs)
1 Answers

SOLUTION this is how insert-if-not-exist can be done in cosmos:

  1. create cosmos client in your code and use ExecuteProcedure to send docs from azure function to cosmos procedure
  2. add stored procedure to your cosmos collection that handles conflicts. You can find bulkUpsert_v_2 from Microsoft and modify it so when conflict 409 occurs, nothing will be written to cosmos.
Related