Pymongo, Motor memory leak

user_7369431 0
Viewed 453

Background: I use tornado + motor, and found the mem_usage increase.

Then I code the test.py. The db.tasks "size" : 12192854 (10+M). After one minute, MEM USAGE / LIMIT is 1.219GiB / 8GiB

env:

  • python 3.7.5
  • motor 2.5.0 (2.1.0 before upgrade)
  • multidict 4.7.5
  • pymongo 3.12.0

Here are my code

import os
import gc
import time
import logging
import asyncio
import uvloop
import pdb
import pymongo
import base64
from tornado.platform.asyncio import AsyncIOMainLoop
from guppy import hpy
from motor import motor_asyncio


mongo_auth = 'xxxxx='
runtime_mongos = arch_mongos = {
    "host": f"mongodb://{base64.b64decode(mongo_auth).decode()}@" + ','.join(
        [
            "1xxx:27024",
            "2xxx:27024",
            "3xxx:27024",
        ]),
    "readPreference": "secondaryPreferred"
}
table = motor_asyncio.AsyncIOMotorClient(**runtime_mongos)["db"]["tasks"]

async def get_data():
    return await table.find().sort([
            ("priority", pymongo.ASCENDING),
            ("start_uts", pymongo.ASCENDING),
        ]).to_list(None)

async def test():
    while True:
        a = await get_data()
        print(len(a))
        await asyncio.sleep(1)
        gc.collect() # no use!

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test())
pymongo tornado python-asyncio python-3.x
1 Answers

Finally, I found the python process has a lot of threads, then I get a clue about the motor 'ThreadPoolExecutor'.

code in motor 2.1:

if 'MOTOR_MAX_WORKERS' in os.environ:
    max_workers = int(os.environ['MOTOR_MAX_WORKERS'])
else:
    max_workers = tornado.process.cpu_count() * 5

_EXECUTOR = ThreadPoolExecutor(max_workers=max_workers)

I set MOTOR_MAX_WORKERS=1 and the mem_usage keeps in low level.

I deploy my project in docker.But, the cpu of the container is not exclusive.I guess this is the reason of 'max_workers' is irrational.

My fault...

user_73694310
Related