I'm using Python3.8, websocket-client==1.1.1, websockets==9.1, unicorn-binance-websocket-api==1.31.0 for work with Binance futures using websockets.
I receive about 600 data objects every 250 ms.
I'm constantly faced with the problem of memory leaks.
The specifics of service:
Every 50 minutes I close the old websocket and open a new websocket.
stream = self.twm.replace_stream(
stream_id=self.streams[ind],
new_channels=channels,
new_markets=markets
)
I used tracemalloc to trace the memory leak. I saw the following picture.
At the start of the process:
[ Top 10 memory usage objects ]
/src/datareader/management/commands/get_data_from_websocket.py:67: size=1656 B, count=5, average=331 B
<frozen importlib._bootstrap>:979: size=648 B, count=2, average=324 B
/src/datareader/management/commands/get_data_from_websocket.py:32: size=576 B, count=1, average=576 B
/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py:40: size=520 B, count=2, average=260 B
/usr/local/lib/python3.8/site-packages/django/core/management/base.py:133: size=272 B, count=2, average=136 B
/src/datareader/management/commands/get_data_from_websocket.py:281: size=184 B, count=2, average=92 B
/src/datareader/management/commands/get_data_from_websocket.py:85: size=184 B, count=2, average=92 B
/src/datareader/management/commands/get_data_from_websocket.py:308: size=136 B, count=1, average=136 B
/src/datareader/management/commands/get_data_from_websocket.py:246: size=136 B, count=1, average=136 B
/src/datareader/management/commands/get_data_from_websocket.py:232: size=136 B, count=1, average=136 B
5 memory blocks: 1.6 KiB
File "/src/datareader/management/commands/get_data_from_websocket.py", line 67
class Command(BaseCommand):
After an hour of work:
[ Top 10 memory usage objects ]
/usr/local/lib/python3.8/site-packages/websockets/legacy/protocol.py:834: size=558 MiB, count=1538074, average=381 B
/usr/local/lib/python3.8/site-packages/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py:783: size=12.8 MiB, count=4, average=3275 KiB
/usr/local/lib/python3.8/site-packages/websockets/extensions/permessage_deflate.py:74: size=2097 KiB, count=64, average=32.8 KiB
<frozen importlib._bootstrap_external>:640: size=236 KiB, count=2342, average=103 B
/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py:1336: size=172 KiB, count=1002, average=176 B
/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py:1340: size=172 KiB, count=1000, average=176 B
/usr/local/lib/python3.8/site-packages/websockets/extensions/permessage_deflate.py:133: size=160 KiB, count=5, average=32.0 KiB
/usr/local/lib/python3.8/abc.py:102: size=135 KiB, count=1301, average=106 B
/usr/local/lib/python3.8/site-packages/django/utils/functional.py:48: size=124 KiB, count=89, average=1429 B
/usr/local/lib/python3.8/site-packages/sentry_sdk/tracing.py:722: size=100 KiB, count=198, average=518 B
1538074 memory blocks: 571569.0 KiB
File "/usr/local/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 834
return frame.data.decode("utf-8") if text else frame.data
After 2 hours of operation:
[ Top 10 memory usage objects ]
/usr/local/lib/python3.8/site-packages/websockets/legacy/protocol.py:834: size=1125 MiB, count=3097554, average=381 B
/usr/local/lib/python3.8/site-packages/unicorn_binance_websocket_api/unicorn_binance_websocket_api_manager.py:783: size=26.0 MiB, count=4, average=6652 KiB
/usr/local/lib/python3.8/site-packages/websockets/extensions/permessage_deflate.py:74: size=3145 KiB, count=90, average=34.9 KiB
/usr/local/lib/python3.8/site-packages/websockets/extensions/permessage_deflate.py:133: size=288 KiB, count=9, average=32.0 KiB
<frozen importlib._bootstrap_external>:640: size=236 KiB, count=2342, average=103 B
/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py:1336: size=172 KiB, count=1001, average=176 B
/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py:1340: size=172 KiB, count=1000, average=176 B
/usr/local/lib/python3.8/site-packages/sentry_sdk/tracing.py:722: size=146 KiB, count=296, average=507 B
/usr/local/lib/python3.8/linecache.py:137: size=146 KiB, count=1423, average=105 B
/usr/local/lib/python3.8/abc.py:102: size=135 KiB, count=1301, average=106 B
3097554 memory blocks: 1151652.4 KiB
File "/usr/local/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 834
return frame.data.decode("utf-8") if text else frame.data
It can be seen that the leak occurs in websockets / legacy / protocol.py
Memory usage graph
Any ideas, suggestions, why this is happening?