I believe I have unearthed a memory leak in my long-lived application when using aiohttp ClientSession requests. If each coroutine which makes a request is awaited sequentially, then all seems fine. However there seems to be a leak of request context manager objects when run concurrently.
Please consider the following example code:
import logging
import tracemalloc
import asyncio
import aiohttp
async def log_allocations_coro():
while True:
await asyncio.sleep(120)
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
str_list = [str(x) for x in top_stats[:5]]
logging.info("\n".join(str_list))
async def do_request():
try:
async with session.request("GET", "http://192.168.1.1") as response:
text = await response.text()
except:
logging.exception("Request failed")
async def main():
tracemalloc.start()
asyncio.ensure_future(log_allocations_coro())
timeout = aiohttp.ClientTimeout(total=1)
global session
session = aiohttp.ClientSession(timeout=timeout)
while True:
tasks = [ do_request(), do_request() ]
await asyncio.gather(*tasks)
await asyncio.sleep(2)
if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s %(message)s', level=logging.INFO)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
The tracemalloc coroutine logs memory allocations every two minutes. This shows the count for allocations in aiohttp/client.py where request() returns a _RequestContextManager increasing over time, initially quickly but then slows until reaching a peak and then seems to be fairly stable.
However, It has then been observed that if there is a network problem and requests start to fail then the count ramps back up again - and doesn't come back down after the problem has been resolved.
Is this a leak? If so, is there a way to work around it?
Thanks for reading!