We have a Rails 6 app that runs on Heroku and yesterday something really bad happened. Our service went down for about 40 minutes, until our manual restart of the dyno fixed the issue.
A method (show) that simply searches an item by ID in the database (a very common query), strangely raised this error:
Rack::Timeout::RequestTimeoutException: Request waited 7ms, then ran for longer than 25000ms
That is very strange since that query normally takes only a few milliseconds (and there was no special load on our website).
After that, all the queries to the database started returning this error:
ActiveRecord::StatementInvalid: PG::DuplicatePstatement: ERROR: prepared statement "a1" already exists
Since we were not doing any change to our app, this means that Heroku performed something unexpected on the database, on the network or on the app that caused a complete disruption of our service...
Basically without a manual restart of the application, the application would be still down.
This is scary because Heroku doesn't restart the dyno automatically in this case and our site keeps returning HTTP 500 to every request because the connection to the database is broken.
Does anyone experienced a similar issue?
Do you know what may have caused this serious issue and how to prevent that?