I am using Python 3 and Django 3.1.4 for a web application hosted on Heroku. I started experiencing R14 and R15 errors related to memory quota exceeding.
The application involves a lot of API calls with large JSON data being sent to the client side via JSONResponse
I followed some online resources to run tracemalloc for some of the views.
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
I refreshed the page and I found this pattern.
Refresh 3:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/decoder.py:353: size=290 MiB, count=3399976, average=89 B
Refresh 4:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/decoder.py:353: size=406 MiB, count=4759963, average=89 B
Refresh 5:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/decoder.py:353: size=522 MiB, count=6119933, average=89 B
Refresh 6:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/decoder.py:353: size=638 MiB, count=7479913, average=89 B
With each refresh, the size increases. It even goes beyond 1000 MiB. Can it be concluded that there is a memory leak with the way JSON data is being handled?
How can I fix this problem?
Update
I ran load testing for all the API calls on the page using loader.io. Did not find any memory leak on Heroku. But as soon as I started browsing the page, it started showing a memory leak.