Concurrent AzureML REST requests fail with Too many requests for service (overloaded)

Question

I have deployed my model to a production Azure Kubernetes Service with 6 nodes.

Sequential inference requests get the expected response from score.py.

When I more than one concurrent async inference requests all the requests except for the first return 503 Too many requests for service {my service name} (overloaded).

I built my service and deployed my model based on the example @ https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb.

I am sending requests as large as 4mb. It seems to work when I send trivially small requests.

user_10664974 · Answer

Check out the guidance here: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-deployment#http-status-code-503

If you're trying to receive many requests concurrently, I recommend you look into high-performance serving with Triton as well: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=python

Concurrent AzureML REST requests fail with Too many requests for service (overloaded)

1 Answers