MongoDB - Error: getMore command failed: Cursor not found

user_3333359 0
Viewed 42356

I need to create a new field sid on each document in a collection of about 500K documents. Each sid is unique and based on that record's existing roundedDate and stream fields.

I'm doing so with the following code:

var cursor = db.getCollection('snapshots').find();
var iterated = 0;
var updated = 0;

while (cursor.hasNext()) {
    var doc = cursor.next();

    if (doc.stream && doc.roundedDate && !doc.sid) {
        db.getCollection('snapshots').update({ "_id": doc['_id'] }, {
            $set: {
                sid: doc.stream.valueOf() + '-' + doc.roundedDate,
            }
        });

        updated++;
    }

    iterated++;
}; 

print('total ' + cursor.count() + ' iterated through ' + iterated + ' updated ' + updated);

It works well at first, but after a few hours and about 100K records it errors out with:

Error: getMore command failed: {
    "ok" : 0,
    "errmsg": "Cursor not found, cursor id: ###",
    "code": 43,
}: ...

mongo error

mongodb-query node.js mongodb database-cursor
6 Answers
user_5339760

I also ran into this problem, but for me it was caused by a bug in the MongDB driver.

It happened in the version 3.0.x of the npm package mongodb which is e.g. used in Meteor 1.7.0.x, where I also recorded this issue. It's further described in this comment and the thread contains a sample project which confirms the bug: https://github.com/meteor/meteor/issues/9944#issuecomment-420542042

Updating the npm package to 3.1.x fixed it for me, because I already had taken into account the good advises, given here by @Danziger.

user_5179140

When using Java v3 driver, noCursorTimeout should be set in the FindOptions.

DBCollectionFindOptions options =
                    new DBCollectionFindOptions()
                        .maxTime(90, TimeUnit.MINUTES)
                        .noCursorTimeout(true)
                        .batchSize(batchSize)
                        .projection(projectionQuery);        
cursor = collection.find(filterQuery, options);
user_12407920

in my case, It was a Load balancing issue, had the same issue running with Node.js service and Mongos as a pod on Kubernetes. The client was using mongos service with default load balancing. changing the kubernetes service to use sessionAffinity: ClientIP (stickiness) resolved the issue for me.

user_43149020

noCursorTimeout will NOT work

now is 2021 year, for

cursor id xxx not found, full error: {'ok': 0.0, 'errmsg': 'cursor id xxx not found', 'code': 43, 'codeName': 'CursorNotFound'}

official says

Consider an application that issues a db.collection.find() with cursor.noCursorTimeout(). The server returns a cursor along with a batch of documents defined by the cursor.batchSize() of the find(). The session refreshes each time the application requests a new batch of documents from the server. However, if the application takes longer than 30 minutes to process the current batch of documents, the session is marked as expired and closed. When the server closes the session, it also kills the cursor despite the cursor being configured with noCursorTimeout(). When the application requests the next batch of documents, the server returns an error.

that means: Even if you have set:

  • noCursorTimeout=True
  • smaller batchSize

will still cursor id not found after default 30 minutes

How to fix/avoid cursor id not found?

make sure two point

  • (explicitly) create new session, get db and collection from this session
  • refresh session periodically

code:

  • (official) js
var session = db.getMongo().startSession()
var sessionId = session.getSessionId().id
var cursor = session.getDatabase("examples").getCollection("data").find().noCursorTimeout()
var refreshTimestamp = new Date() // take note of time at operation start
while (cursor.hasNext()) {
  // Check if more than 5 minutes have passed since the last refresh
  if ( (new Date()-refreshTimestamp)/1000 > 300 ) {
    print("refreshing session")
    db.adminCommand({"refreshSessions" : [sessionId]})
    refreshTimestamp = new Date()
  }
  // process cursor normally
}
  • (mine) python
import logging
from datetime import datetime
import pymongo

mongoClient = pymongo.MongoClient('mongodb://127.0.0.1:27017/your_db_name')

# every 10 minutes to update session once
#   Note: should less than 30 minutes = Mongo session defaul timeout time
#       https://docs.mongodb.com/v5.0/reference/method/cursor.noCursorTimeout/
# RefreshSessionPerSeconds = 10 * 60
RefreshSessionPerSeconds = 8 * 60

def mergeHistorResultToNewCollection():

    mongoSession = mongoClient.start_session() # <pymongo.client_session.ClientSession object at 0x1081c5c70>
    mongoSessionId = mongoSession.session_id # {'id': Binary(b'\xbf\xd8\xd...1\xbb', 4)}

    mongoDb = mongoSession.client["your_db_name"] # Database(MongoClient(host=['127.0.0.1:27017'], document_class=dict, tz_aware=False, connect=True), 'your_db_name')
    mongoCollectionOld = mongoDb["collecion_old"]
    mongoCollectionNew = mongoDb['collecion_new']

    # historyAllResultCursor = mongoCollectionOld.find(session=mongoSession)
    historyAllResultCursor = mongoCollectionOld.find(no_cursor_timeout=True, session=mongoSession)

    lastUpdateTime = datetime.now() # datetime.datetime(2021, 8, 30, 10, 57, 14, 579328)
    for curIdx, oldHistoryResult in enumerate(historyAllResultCursor):
        curTime = datetime.now() # datetime.datetime(2021, 8, 30, 10, 57, 25, 110374)
        elapsedTime = curTime - lastUpdateTime # datetime.timedelta(seconds=10, microseconds=531046)
        elapsedTimeSeconds = elapsedTime.total_seconds() # 2.65892
        isShouldUpdateSession = elapsedTimeSeconds > RefreshSessionPerSeconds
        # if (curIdx % RefreshSessionPerNum) == 0:
        if isShouldUpdateSession:
            lastUpdateTime = curTime
            cmdResp = mongoDb.command("refreshSessions", [mongoSessionId], session=mongoSession)
            logging.info("Called refreshSessions command, resp=%s", cmdResp)
        
        # do what you want

        existedNewResult = mongoCollectionNew.find_one({"shortLink": "http://xxx"}, session=mongoSession)

    # mongoSession.close()
    mongoSession.end_session()
user_16162630
Related