MongoDB: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction

user_1327093 0
Viewed 6832

Getting WriteConflict error frequently when calling an api in parallel(4 times) that does new inserts of batchsize of 10K docs in a transaction. But if the insert batchsize is set to 2K it runs fine. not sure which parameter I need to adjust so that insertion works without issue for 10k batchsize

MongoDB server version: 4.4.0

Not sure if this has sth to do with this https://jira.mongodb.org/browse/SERVER-53464

Here is the mongo log for same when mongo loglevel set to 1

    {"t":{"$date":"2021-01-23T17:23:40.838+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn42","msg":"Slow query","attr":{"type":"command","ns":"mydb.mycollection","command":{"insert":"mycollection","ordered":false,"$db":"mydb","lsid":{"id":{"$uuid":"a0beb5f8-ed4c-4b64-8971-719178b3a598"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1611422612,"i":10000}},"signature":{"hash":{"$binary":{"base64":"H3DsiEiy4CMrRNvWqbIJQMbHl0c=","subType":"0"}},"keyId":6870929147875557381}},"txnNumber":4,"startTransaction":true,"autocommit":false},"ninserted":10000,"keysInserted":30000,"numYields":0,"reslen":230,"locks":{},"storage":{"data":{"bytesRead":1661221,"timeReadingMicros":3201739}},"protocol":"op_msg","durationMillis":3741}}
{"t":{"$date":"2021-01-23T17:23:40.838+00:00"},"s":"D1", "c":"QUERY",    "id":22790,   "ctx":"conn42","msg":"Received interrupt request for unknown op","attr":{"opId":28178}}
{"t":{"$date":"2021-01-23T17:23:40.909+00:00"},"s":"I",  "c":"TXN",      "id":51802,   "ctx":"conn42","msg":"transaction","attr":{"parameters":{"lsid":{"id":{"$uuid":"a0beb5f8-ed4c-4b64-8971-719178b3a598"},"uid":{"$binary":{"base64":"Zx/+1Rn/RW/eLwwG9xqjQAzGuq4fRkYljdz5CIhAbaw=","subType":"0"}}},"txnNumber":4,"autocommit":false,"readConcern":{"provenance":"implicitDefault"}},"readTimestamp":"Timestamp(0, 0)","ninserted":10000,"keysInserted":30000,"terminationCause":"aborted","timeActiveMicros":3810904,"timeInactiveMicros":2702,"numYields":0,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":2}},"Global":{"acquireCount":{"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":46}}},"storage":{"data":{"bytesRead":1661221,"timeReadingMicros":3201739}},"wasPrepared":false,"durationMillis":3813}}
{"t":{"$date":"2021-01-23T17:23:40.910+00:00"},"s":"D1", "c":"STORAGE",  "id":22411,   "ctx":"conn42","msg":"Slow WT transaction. Lifetime of SnapshotId {getSnapshotId_toNumber} was {transactionTime}ms","attr":{"getSnapshotId_toNumber":78620,"transactionTime":3813}}
{"t":{"$date":"2021-01-23T17:23:40.933+00:00"},"s":"D1", "c":"COMMAND",  "id":21962,   "ctx":"conn42","msg":"Assertion while executing command","attr":{"command":"commitTransaction","db":"admin","commandArgs":{"commitTransaction":1,"$db":"admin","lsid":{"id":{"$uuid":"a0beb5f8-ed4c-4b64-8971-719178b3a598"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1611422620,"i":10000}},"signature":{"hash":{"$binary":{"base64":"+r9TOIIR/J8dXRvrPyF0urUeWJA=","subType":"0"}},"keyId":6870929147875557381}},"txnNumber":4,"autocommit":false},"error":"WriteConflict: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction."}}
{"t":{"$date":"2021-01-23T17:23:40.933+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn42","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"commitTransaction":1,"$db":"admin","lsid":{"id":{"$uuid":"a0beb5f8-ed4c-4b64-8971-719178b3a598"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1611422620,"i":10000}},"signature":{"hash":{"$binary":{"base64":"+r9TOIIR/J8dXRvrPyF0urUeWJA=","subType":"0"}},"keyId":6870929147875557381}},"txnNumber":4,"autocommit":false},"numYields":0,"ok":0,"errMsg":"WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.","errName":"WriteConflict","errCode":112,"reslen":394,"locks":{"ReplicationStateTransition":{"acquireCount":{"w":3}},"Global":{"acquireCount":{"r":1,"w":1}},"Database":{"acquireCount":{"w":1}},"Collection":{"acquireCount":{"w":1}},"Mutex":{"acquireCount":{"r":46}}},"flowControl":{"acquireCount":1,"timeAcquiringMicros":1},"storage":{},"protocol":"op_msg","durationMillis":93}}
{"t":{"$date":"2021-01-23T17:23:40.933+00:00"},"s":"D1", "c":"QUERY",    "id":22790,   "ctx":"conn42","msg":"Received interrupt request for unknown op","attr":{"opId":28235}}
mongodb
2 Answers

From the mongodb transaction docs:

Number of operations in a transaction: There are no hard limits to the number of documents that can be read within a transaction. As a best practice, no more than 1,000 documents should be modified within a transaction.

For operations that need to modify more than 1,000 documents, developers should break the transaction into separate parts that process documents in batches.

user_104150470

You may have missed putting await in an async task; e.g.,

await place.remove({ session: sess });
user_128543960
Related