Most efficient MongoDB find() query for very large data

I trying to implement a research system for a scientific study in 2D astronomical coordinate systems. Into an hand I have a process that generates a lot of geospatial data that we can thing organized in documents of two coordinates and a string value. In the other hand I have a little set of data which have only one coordinate that must match wit at least one of the two of the generated date.

Simplifying, I have organized data in two collections:

• collA avg. size: ~2GB (constant in the time)
• collB more than 50GB (continuously increasing)

Where:

The schema of a document in the collA is:

{
    terrainType: 'myType00001',
    lat: '000000123',
    lon: '987000000'
},
{
    terrainType: 'myType00002',
    lat: '000000124',
    lon: '987000000'
},
{
    terrainType: 'myType00003',
    lat: '000000124',
    lon: '997000000'
}

Please note that first of all we put indexes to avoid COLLSCAN. There are two indexes in collA: __lat_idx (unique) and __long_idx (unique). I can guarantee that the generation process could not generate duplicates in lat e in lot columns (as seen above: lat and lon have nine digits, but it is only for simplicty... in the real case these values are extremely huge).

The schema of a document in the collB is:

{
    latOrLon: '0045600'
},
{
    latOrLon: '0045622'
},
{
    latOrLon: '1145600'
}

I tried some different query strategies.

Strategy A

let cursor = collB.find() // Should load entire 2GB into memory?

curosor.forEach(c => {
    collA.find({lat: c.latOrLon})
    collA.find({lon: c.latOrLon})
})

This takes two mongo calls for each document in collB: is extremely slow.

Strategy B

let cursor = collB.find() // Should load entire 2GB into memory?

curosor.forEach(c => {
    collA.find({$expr: {$or: [{$eq: [lat, c.latOrLon]}, {$eq: [lon, c.latOrLon]} })
})

This taks on mongo call for each document in collB, faster than A but still slow.

Strategy C

for chunk in chunks:
    docs = []
    for doc in chunk:
    CHUNK_SIZE = 5000
    batch_cursor = collB.find({}, {'_id': 0}, batch_size=CHUNK_SIZE)
    chunks = yield_rows(batch_cursor, CHUNK_SIZE) # yield_rows defined below
    res = collA.find({
                    '$or': [
                        {
                            'lon': {
                                '$in': docs
                            }
                        }, {
                            'lat': {
                                '$in': docs
                            }
                        }
                    ]
                })

This takes firstly 5000 documents from collA then puts them into an array and send the find() query. Good efficiency: one mongo call for each 5000 documents in collA.

This solution is very fast, but I noticed that becomes slower as collA increases in size, why? Since I am using indexes, the search for a indexed value should costs O(1) in terms of computation time... for example, when collA was around 25GB in size, it will takes roughly 30 minutes to perform a full find(). Now this collection is 50GB size and it is taking morte than 2 hours.

We plan that the DB will reach at least 5TB next month and this will be a problem.

I am asking to university the possibility to parallelize this job using MongoDB Sharding but it will no be immediate. I am looking for a temporary solution until we can parallelize this job.

Please note that we tried more than these three strategies and we mixed Python3 and NodeJS.

Definition of yield_rows:

        def yield_rows(batch_cursor, chunk_size):
            """
            Generator to yield chunks from cursor
            :param cursor:
            :param chunk_size:
            :return:
            """
            chunk = []
            for i, row in enumerate(batch_cursor):
                if i % chunk_size == 0 and i > 0:
                    yield chunk
                    del chunk[:]
                chunk.append(row)
            yield chunk