I have been trying to get a containerized machine learning model to work on AWS sagemaker through its batch transform service which breaks the entire dataset into smaller data sets for inference from the machine learning model.
The container has a flask service which runs the ML model with gunicorn and nginx in the background. While executing the batch transform i get the 502 bad gateway error with following error on logs (when i ran the same container with 50k dataset as input it passed with a c5.xlarge instance but failed when i ran with same circumstances under 80k)
*4 upstream prematurely closed connection while reading response header from
upstream, client: IP, server: , request: "POST /invocations
HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/invocations", host:
"IP:8080"
"POST /invocations HTTP/1.1" 502 182 "-" "Apache-HttpClient/4.5.x (Java/1.8.0_172)"
Nginx Configuration
worker_processes 1;
daemon off; # Prevent forking
pid /tmp/nginx.pid;
error_log /var/log/nginx/error.log;
events { defaults }
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log combined;
upstream gunicorn {
server unix:/tmp/gunicorn.sock;
}
server {
listen 8080 deferred;
client_max_body_size 5m;
keepalive_timeout 10000;
location ~ ^/(ping|invocations) {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://gunicorn;
}
location / {
return 404 "{}";
}
}
}
and gunicorn config:
I am quite new to nginx and gunicorn and have read through most of the other so posts with upstream prematurely closed with connection while reading response error. I have tried somethings like increasing the client body size etc but still ran into same error. Some help with regards to this will be really helpful.