I run a Flask application on Cloud Run with gunicorn and worker type gthread. On low level, this API just writes/retrieves entries from Datastore and Pubsub so all operations are light. The traffic is 20,000 requests a day with max QPS 20 requests and average QPS 2
Every few days/weeks, a portion of traffic (25%-50%) gets a 504 response with the following error:
The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout
After a few hours, the issue resolves itself (redeploying the app or a new CR instance kicking in solves it instantly). Those 504 requests don't seem to even reach the application. This error does not seem to be linked to any errors in stderr, stdout or increases in QPS. I've set up some advanced logging and cloud profiler but I don't get any useful information. This application ran perfectly fine on App Engine with gunicorn sync worker. The only difference is gunicorn settings. Sync worker worked very slow on CR so I used gthread instead.
Any ideas where to look next?
Dockerfile
ENTRYPOINT ["gunicorn"]
CMD ["-b", ":8080", "--worker-class", "gthread", "--workers", "5", "--threads", "2", "api.app:app" ]
app.py
def run(**kwargs):
app.run(port=8080, debug=False)
app = connexion.App(__name__, specification_dir='./swagger/')
app.app.json_encoder = encoder.JSONEncoder
app.add_api('swagger.yaml', resolver=connexion.resolver.RestyResolver('api'))
app.app.config.from_object(Config)
if __name__ == '__main__':
run()
gcloud deploy command:
gcloud run deploy ${service_name} \
...
--image=${image} \
--concurrency=80 \
--memory=2Gi \
--cpu=2
...