Python aiohttp ClientSession requests memory leak?

Viewed 697

I believe I have unearthed a memory leak in my long-lived application when using aiohttp ClientSession requests. If each coroutine which makes a request is awaited sequentially, then all seems fine. However there seems to be a leak of request context manager objects when run concurrently.

Please consider the following example code:

import logging
import tracemalloc
import asyncio
import aiohttp


async def log_allocations_coro():

    while True:
        await asyncio.sleep(120)
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics('lineno')
        str_list = [str(x) for x in top_stats[:5]]
        logging.info("\n".join(str_list))


async def do_request():

    try:
        async with session.request("GET", "http://192.168.1.1") as response:
            text = await response.text()
    except:
        logging.exception("Request failed")


async def main():

    tracemalloc.start()
    asyncio.ensure_future(log_allocations_coro())
    
    timeout = aiohttp.ClientTimeout(total=1)
    global session
    session = aiohttp.ClientSession(timeout=timeout)

    while True:

        tasks = [ do_request(), do_request() ]
        await asyncio.gather(*tasks)
        await asyncio.sleep(2)


if __name__ == '__main__':

    logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

The tracemalloc coroutine logs memory allocations every two minutes. This shows the count for allocations in aiohttp/client.py where request() returns a _RequestContextManager increasing over time, initially quickly but then slows until reaching a peak and then seems to be fairly stable. However, It has then been observed that if there is a network problem and requests start to fail then the count ramps back up again - and doesn't come back down after the problem has been resolved.

Is this a leak? If so, is there a way to work around it?

Thanks for reading!

0 Answers
Related