Heroku Python App Suddenly Giving R15 Error

Question

I'm using free heroku dynos and postgres, running a python flask app. It often cut it close on the memory limits of free dynos (the app loads a few models upon startup), but for eight months it was running no problem. Then today, I saw this error:

2021-04-15T08:28:41.566146+00:00 heroku[web.1]: State changed from crashed to starting
2021-04-15T08:28:55.333917+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2021-04-15T08:28:57.226420+00:00 app[web.1]: [2021-04-15 08:28:57 +0000] [4] [INFO] Starting gunicorn 19.9.0
2021-04-15T08:28:57.226786+00:00 app[web.1]: [2021-04-15 08:28:57 +0000] [4] [INFO] Listening at: http://0.0.0.0:7490 (4)
2021-04-15T08:28:57.226838+00:00 app[web.1]: [2021-04-15 08:28:57 +0000] [4] [INFO] Using worker: sync
2021-04-15T08:28:57.229746+00:00 app[web.1]: [2021-04-15 08:28:57 +0000] [10] [INFO] Booting worker with pid: 10
2021-04-15T08:28:57.313381+00:00 app[web.1]: [2021-04-15 08:28:57 +0000] [11] [INFO] Booting worker with pid: 11
2021-04-15T08:28:58.983920+00:00 heroku[web.1]: State changed from starting to up
2021-04-15T08:29:01.760373+00:00 heroku[web.1]: Process running mem=512M(100.0%)
2021-04-15T08:29:01.788242+00:00 app[web.1]: [2021-04-15 08:29:01 +0000] [42] [INFO] Booting worker with pid: 42
2021-04-15T08:29:01.800651+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2021-04-15T08:29:01.802395+00:00 heroku[web.1]: Stopping process with SIGKILL
2021-04-15T08:29:01.912435+00:00 heroku[web.1]: Process exited with status 137
2021-04-15T08:29:01.984054+00:00 heroku[web.1]: State changed from up to crashed
2021-04-15T08:30:57.809476+00:00 app[api]: Starting process with command `ps:scale web=2` by user large.adam.m@gmail.com
2021-04-15T08:31:14.882047+00:00 heroku[run.4863]: Awaiting client
2021-04-15T08:31:14.899189+00:00 heroku[run.4863]: Starting process with command `ps:scale web=2`
2021-04-15T08:31:15.029551+00:00 heroku[run.4863]: State changed from starting to up
2021-04-15T08:31:19.037705+00:00 heroku[run.4863]: Process exited with status 127
2021-04-15T08:31:19.083172+00:00 heroku[run.4863]: State changed from up to complete
2021-04-15T08:33:16.656240+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=ludothemes.herokuapp.com request_id=17839d00-978a-4e4e-b188-cb1570d871f7 fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http
2021-04-15T08:33:22.257162+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=ludothemes.herokuapp.com request_id=f7a25e76-f51c-479a-a922-4a5dce350d5c fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http
2021-04-15T08:36:21.885333+00:00 heroku[web.1]: State changed from crashed to starting
2021-04-15T08:36:34.035395+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=ludothemes.herokuapp.com request_id=381addaf-a75b-4c7a-9309-126029119c4f fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http
2021-04-15T08:36:39.535412+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=ludothemes.herokuapp.com request_id=a8f0edf8-87e6-4a8a-8aa6-4f83e01661df fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http
2021-04-15T08:36:40.641472+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2021-04-15T08:36:42.379884+00:00 app[web.1]: [2021-04-15 08:36:42 +0000] [4] [INFO] Starting gunicorn 19.9.0
2021-04-15T08:36:42.380270+00:00 app[web.1]: [2021-04-15 08:36:42 +0000] [4] [INFO] Listening at: http://0.0.0.0:43156 (4)
2021-04-15T08:36:42.380320+00:00 app[web.1]: [2021-04-15 08:36:42 +0000] [4] [INFO] Using worker: sync
2021-04-15T08:36:42.383371+00:00 app[web.1]: [2021-04-15 08:36:42 +0000] [10] [INFO] Booting worker with pid: 10
2021-04-15T08:36:42.429343+00:00 app[web.1]: [2021-04-15 08:36:42 +0000] [11] [INFO] Booting worker with pid: 11
2021-04-15T08:36:42.740845+00:00 heroku[web.1]: State changed from starting to up
2021-04-15T08:36:46.581577+00:00 heroku[web.1]: Process running mem=512M(100.0%)
2021-04-15T08:36:46.607496+00:00 app[web.1]: [2021-04-15 08:36:46 +0000] [42] [INFO] Booting worker with pid: 42
2021-04-15T08:36:46.632959+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2021-04-15T08:36:46.635253+00:00 heroku[web.1]: Stopping process with SIGKILL
2021-04-15T08:36:46.702243+00:00 heroku[web.1]: Process exited with status 137
2021-04-15T08:36:46.883019+00:00 heroku[web.1]: State changed from up to crashed
2021-04-15T08:44:56.645111+00:00 heroku[web.1]: State changed from crashed to starting
2021-04-15T08:45:09.196127+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2021-04-15T08:45:10.572619+00:00 app[web.1]: [2021-04-15 08:45:10 +0000] [4] [INFO] Starting gunicorn 19.9.0
2021-04-15T08:45:10.572997+00:00 app[web.1]: [2021-04-15 08:45:10 +0000] [4] [INFO] Listening at: http://0.0.0.0:44998 (4)
2021-04-15T08:45:10.573055+00:00 app[web.1]: [2021-04-15 08:45:10 +0000] [4] [INFO] Using worker: sync
2021-04-15T08:45:10.575990+00:00 app[web.1]: [2021-04-15 08:45:10 +0000] [10] [INFO] Booting worker with pid: 10
2021-04-15T08:45:10.672926+00:00 app[web.1]: [2021-04-15 08:45:10 +0000] [11] [INFO] Booting worker with pid: 11
2021-04-15T08:45:10.792978+00:00 heroku[web.1]: State changed from starting to up
2021-04-15T08:45:14.493337+00:00 heroku[web.1]: Process running mem=512M(100.0%)
2021-04-15T08:45:14.513393+00:00 app[web.1]: [2021-04-15 08:45:14 +0000] [42] [INFO] Booting worker with pid: 42
2021-04-15T08:45:14.538602+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2021-04-15T08:45:14.539922+00:00 heroku[web.1]: Stopping process with SIGKILL
2021-04-15T08:45:14.549012+00:00 heroku[web.1]: Process running mem=264M(51.6%)
2021-04-15T08:45:14.636653+00:00 heroku[web.1]: Process exited with status 137
2021-04-15T08:45:14.682557+00:00 heroku[web.1]: State changed from up to crashed
2021-04-15T08:45:14.691912+00:00 heroku[web.1]: State changed from crashed to starting
2021-04-15T08:45:33.819016+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2021-04-15T08:45:35.480103+00:00 app[web.1]: [2021-04-15 08:45:35 +0000] [4] [INFO] Starting gunicorn 19.9.0
2021-04-15T08:45:35.480524+00:00 app[web.1]: [2021-04-15 08:45:35 +0000] [4] [INFO] Listening at: http://0.0.0.0:5640 (4)
2021-04-15T08:45:35.480582+00:00 app[web.1]: [2021-04-15 08:45:35 +0000] [4] [INFO] Using worker: sync
2021-04-15T08:45:35.483462+00:00 app[web.1]: [2021-04-15 08:45:35 +0000] [10] [INFO] Booting worker with pid: 10
2021-04-15T08:45:35.568432+00:00 app[web.1]: [2021-04-15 08:45:35 +0000] [11] [INFO] Booting worker with pid: 11
2021-04-15T08:45:36.044752+00:00 heroku[web.1]: State changed from starting to up
2021-04-15T08:45:41.085020+00:00 heroku[web.1]: Process running mem=512M(100.0%)
2021-04-15T08:45:41.116608+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2021-04-15T08:45:41.118394+00:00 heroku[web.1]: Stopping process with SIGKILL
2021-04-15T08:45:41.118709+00:00 app[web.1]: [2021-04-15 08:45:41 +0000] [42] [INFO] Booting worker with pid: 42
2021-04-15T08:45:41.135141+00:00 heroku[web.1]: Process running mem=244M(47.7%)
2021-04-15T08:45:41.171322+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2021-04-15T08:45:41.175932+00:00 heroku[web.1]: Stopping process with SIGKILL
2021-04-15T08:45:41.201559+00:00 heroku[web.1]: Process exited with status 137
2021-04-15T08:45:41.251165+00:00 heroku[web.1]: State changed from up to crashed
2021-04-15T08:52:00.055423+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=ludothemes.herokuapp.com request_id=e9db949e-c3db-4cbe-b770-2ee66deabc8d fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http
2021-04-15T08:52:05.682515+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=ludothemes.herokuapp.com request_id=dd0f05d4-a907-4f1b-b51f-15a7374e96ab fwd="107.212.209.19" dyno= connect= service= status=503 bytes= protocol=http

I get R15 errors, but it seems to not get to twice the limits of the free dynos. Did something change in heroku or python that would affect how my app is loading all of a sudden?

Any suggestions of things to look into would be appreciated!

user_9829074 · Answer

Exactly the same problem for me - my app was running for several months using free dynos, there was no problem going at 180% memory usage, but since 2 days ago it's failing:

Process running mem=512M(100.0%)
Error R15 (Memory quota vastly exceeded)
Stopping process with SIGKILL

I was hopeful that this incident might be related, but unfortunately they have closed it.

user_4937720 · Answer

This seems to have been an issue with Heroku and to now be fixed. I've sent Heroku a ticket and received the following reply yesterday:

It looks like this was related to a new infrastructure rollout our engineering team is conducting. They've recently put in a fix to correctly calculate R15 errors correctly so you shouldn't experience any additional issues. We apologize for any inconvenience this has caused, please let us know if you have any questions.

I confirm that things are working on my app at least. Hope it's the same to you.

Heroku Python App Suddenly Giving R15 Error

2 Answers