GridFS Large File Download Crashes the Node.js Server. MongoServerError: Sort exceeded memory limit of bytes

user_10850718 0
Viewed 79

So I'm running a Node.js server with mongoose and I've been trying to download a large file using the GridFS node package but I've been getting the following error:

MongoServerError: Executor error during find command :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.

Here is the code that I've been using to download the file:

// Stream a single file. 
router.get('/stream/:filename', async function(req, res, next){
    try{        
        const file = gfs.find({
            filename: req.params.filename
        }).toArray((err, files) => {
        if (!files || files.length === 0) {
            return res.status(404)
            .json({
                err: "no files exist"
            });
        }

        gfs.openDownloadStreamByName(req.params.filename)
            .pipe(res);
        });
    }catch(err){
        return res.status(400).json({
            err:'Could not fetch the file.'
        })
    }
});

Given that it's OK to answer your own question on StackOverflow, I’d like to document this problem so that my solution may help others encountering it later.

gridfs node.js mongodb indexing file
1 Answers

As it turns out, when you download a file by streaming it with GridFS, the documents that compose the file are sorted. According to this blog post, when doing a sort, MongoDB first attempts to retrieve the documents using the order specified in an index. When no index is available it will try to load the documents into memory and sort them there.

The catch is that Mongo is configured by default to abort the operation when exceeding usage of 32 MB. This is why we run into the “Sort exceeded memory limit” error described above. In order to solve this problem then, you’ll have to create an index for the ‘n’ field that contains the order of the file chunks that you want to download. To do this you can either create the index directly through your database interface (I use the MongoDB Compass GUI), or you can create it through code using the createIndex() method:

// Create an index for the 'n' field to help sort the chunks collection.
db.collection('files.chunks').createIndex({n: 1});
user_108507180
Related