sagemaker batch transform breaks with upstream prematurely closed connection while reading upstream

Viewed 3053

I have been trying to get a containerized machine learning model to work on AWS sagemaker through its batch transform service which breaks the entire dataset into smaller data sets for inference from the machine learning model.

The container has a flask service which runs the ML model with gunicorn and nginx in the background. While executing the batch transform i get the 502 bad gateway error with following error on logs (when i ran the same container with 50k dataset as input it passed with a c5.xlarge instance but failed when i ran with same circumstances under 80k)

*4 upstream prematurely closed connection while reading response header from 
upstream, client: IP, server: , request: "POST /invocations 
HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/invocations", host: 
"IP:8080"

"POST /invocations HTTP/1.1" 502 182 "-" "Apache-HttpClient/4.5.x (Java/1.8.0_172)"

Nginx Configuration

worker_processes 1;
daemon off; # Prevent forking
pid  /tmp/nginx.pid;
error_log /var/log/nginx/error.log;
events { defaults }
http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    access_log /var/log/nginx/access.log combined;

    upstream gunicorn {
        server unix:/tmp/gunicorn.sock;
    }

    server {
       listen 8080 deferred;
       client_max_body_size 5m;

       keepalive_timeout 10000;

       location ~ ^/(ping|invocations) {
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header Host $http_host;
           proxy_redirect off;
           proxy_pass http://gunicorn;
      }

     location / {
       return 404 "{}";
     }
  } 
}

and gunicorn config:

https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/container/decision_trees/serve

I am quite new to nginx and gunicorn and have read through most of the other so posts with upstream prematurely closed with connection while reading response error. I have tried somethings like increasing the client body size etc but still ran into same error. Some help with regards to this will be really helpful.

1 Answers

This looks like a gunicorn worker timeout. There are two timeout settings you can adjust depending on the time it takes for your model to serve inference requests:

  1. The gunicorn worker timeout, which can be adjusted here: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/container/decision_trees/serve#L25

  2. The nginx proxy_read_timeout setting, which can be added to the nginx.conf here: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/container/decision_trees/nginx.conf#L21-L37

If you require support for specific transform jobs, please reach out on AWS forums: https://forums.aws.amazon.com/forum.jspa?forumID=285&start=0

Related