ElasticBeanstalk nginx returning 400 Bad Request when Context-Length header is sent

user_6199978 0
Viewed 1198

We are trying our first .Net Core 3.0 web app in a Docker container. We are deploying through AWS ElasticBeanstalk to an Amazon Linux 2 AMI. We are able to do any GET requests of the server without an issue. Anything with a body in the Request fails with a 400 Bad Request error. Both PATCH and POST fail. PATCH works if the endpoint does not expect a body. The content we are sending is only 147 bytes, so it should not even be hitting the default client_max_body_size of 1m.

I found that if I remove the Content-Length header in my Postman test the request gets through to the server, but, it seems no Body is included.

I've tested locally running from both VisualStudio and from Docker Desktop and it works. My local OS is Windows 10 though, not, Linux, and I'm not using nginx or IIS locally.

I've tried setting my client_max_body_size to 0 (and 20m) by putting a file named nginxext.conf in ".platform/nginx/conf.d". I'm pretty sure it's being read because I got an error the first time because the file was not UTF-8 encoded.

I know I'm missing something really basic. This just doesn't seem like an off the wall configuration, using .Net Core 3.0 in Docker on Linux for a website that accepts POSTs. ??? Let alone posting something with a body.

Thanks in advance,

Side question: Does anyone know where I can find the "default" nginx configuration used by AWS for their Amazon Linux 2 images?

nginx-reverse-proxy docker .net-core nginx
2 Answers

I ended up requesting support from AWS. After going back and forth and reviewing some links he provided we were able to solve the issue.

https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-3.1 https://github.com/aspnet/KestrelHttpServer/issues/1263

I ended up with a full nginx.conf file with my deployment, but, I'm pretty sure these lines were what fixed the issue for me.

  http
    server
        location / {
            proxy_pass                                  http://docker;
            proxy_http_version                          1.1;
            proxy_set_header    Connection              $http_connection;
            proxy_set_header    Upgrade                 $http_upgrade;
            proxy_set_header    Host                    $host;
            proxy_set_header    X-Real-IP               $remote_addr;
            proxy_set_header    X-Forwarded-Host        $http_host;
            proxy_set_header    X-Forwarded-For         $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Proto       $scheme;
            proxy_cache_bypass                          $http_upgrade;

            client_max_body_size 100M;

I also added these lines to the very beginning of my Startup.Configuration() method.

using Microsoft.AspNetCore.HttpOverrides;
        
        app.UseForwardedHeaders(new ForwardedHeadersOptions
        {
            ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto
        });

Thank you Ali P (AWS Support) and @Marcin for your help.

Full nginx.conf file

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
# Elastic Beanstalk Nginx Configuration File ( nginx  for my app ) :

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
worker_rlimit_nofile    32136;

events {
    worker_connections 2048;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log    /var/log/nginx/access.log;


    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

    include  conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
            default       "upgrade";
    }

    server {
        listen 80 default_server;
        gzip on;
        gzip_comp_level 4;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        access_log    /var/log/nginx/access.log main;

        location / {
            proxy_pass                                  http://docker;
            proxy_http_version                          1.1;
            #proxy_set_header    Connection              $connection_upgrade;
            #proxy_set_header    Connection keep-alive;
            #proxy_set_header    Connection              "Upgrade";
            proxy_set_header    Connection              $http_connection;
            proxy_set_header    Upgrade                 $http_upgrade;
            proxy_set_header    Host                    $host;
            proxy_set_header    X-Real-IP               $remote_addr;
            proxy_set_header    X-Forwarded-Host        $http_host;
            proxy_set_header    X-Forwarded-For         $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Proto       $scheme;
            proxy_cache_bypass                          $http_upgrade;

            client_max_body_size 100M;

        }

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;
    }
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:
upstream docker {
    server 172.17.0.3:80;
    keepalive 256;
}
# configuration file /etc/nginx/conf.d/healthd_logformat.conf:
log_format healthd  '$msec"$uri"'
                    '$status"$request_time"$upstream_response_time"'
                    '$http_x_forwarded_for';
# configuration file /etc/nginx/conf.d/elasticbeanstalk/healthd.conf:
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
    set $year $1;
    set $month $2;
    set $day $3;
    set $hour $4;
}

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
user_61999780

Based on the comments.

I checked the default nginx config file using sample application on Docker running on 64bit Amazon Linux 2/3.1.0.

/etc/nginx/nginx.conf

# Elastic Beanstalk Nginx Configuration File

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
worker_rlimit_nofile    32136;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log    /var/log/nginx/access.log;


    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

    include  conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
            default       "upgrade";
    }

    server {
        listen 80 default_server;
        gzip on;
        gzip_comp_level 4;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        access_log    /var/log/nginx/access.log main;

        location / {
            proxy_pass            http://docker;
            proxy_http_version    1.1;

            proxy_set_header    Connection             $connection_upgrade;
            proxy_set_header    Upgrade                $http_upgrade;
            proxy_set_header    Host                   $host;
            proxy_set_header    X-Real-IP              $remote_addr;
            proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
        }

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;
    }

/etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf

upstream docker {
    server 172.17.0.2:8000;
    keepalive 256;
}

/etc/nginx/conf.d/healthd_logformat.conf

log_format healthd  '$msec"$uri"'
                    '$status"$request_time"$upstream_response_time"'
                    '$http_x_forwarded_for';[ec2-user@ip-172-31-36-64 conf.d]$ readlink -f ./healthd_logformat.conf

/etc/nginx/conf.d/elasticbeanstalk/healthd.conf

if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
    set $year $1;
    set $month $2;
    set $day $3;
    set $hour $4;
}
user_2488230
Related