haproxy compression and chunked encoding

Question

We detected a very strange bug in haproxy 1.7.9 today, that we are using from a Centos Build:

When we add the compression keyword and the transfer is chunked because of the size of the file, no matter if we Accept-Encoding: gzip (or whatever), or not, the haproxy does not finish the transfer correctly but hangs forever.

Removing all compression keyword lines from the haproxy configuration works perfectly though, because the backend application server already parses the Accept-Encoding header for compression algorithms and compresses the content itself.

I would thus mark the compression keyword in haproxy as dangerous. This situation is very hard to debug and its even harder to figure out, why haproxy behaves that bad!

Is there any version where this bug is removed?
Why does this even happen with compression algo identity?
Why does this happen, though compression should be turned off for chunked transfers?
Why does this happens though the mime-type is not within the compression type set of mime-types?

(Bonus Question: How can any professional live with such bad situations?)

TERMINAL OUTPUT FROM THE TESTS:

admin@bastion1$ time curl -v -k -o/dev/null -H "Accept-Encoding: gzip" 'http://IP/URLto/deps.js'
* About to connect() to XXX port 80 (#0)
*   Trying XXX...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to XXX (XXX) port 80 (#0)
> GET /URLto/deps.js HTTP/1.1
> User-Agent: curl/7.29.0
> Host: XXX
> Accept: */*
> Accept-Encoding: gzip
>
< HTTP/1.1 200 OK
< Connection: close
< Date: Wed, 09 Oct 2019 15:14:29 GMT
< Last-Modified: Mon, 02 Sep 2019 19:28:26 GMT
< ETag: "3bfdf549feec78af317650f9c35a7687"
< Content-Type: application/javascript
< Vary: Accept-Encoding
<
{ [data not shown]
100 15136    0 15136    0     0    247      0 --:--:--  0:01:01 --:--:--     0^C

ends in a timeout, with or without Accept-Encoding: gzip and this also happens if of if not compression type is application/javascript or just something else and even with compression offload.

Removing all compression keywords in the haproxy sections enables the application server to do its compression on its own and delivers the data as requested by Accept-Encoding:

admin@bastion1$ time curl -v -k -o/dev/null -H "Accept-Encoding: gzip" 'http://.../deps.js' [...]
> GET .../deps.js HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Accept-Encoding: gzip
>
< HTTP/1.1 200 OK
< Connection: close
< Date: Wed, 09 Oct 2019 15:20:40 GMT
< Last-Modified: Mon, 02 Sep 2019 19:28:26 GMT
< ETag: "3bfdf549feec78af317650f9c35a7687--gzip"
< Content-Type: application/javascript
< Vary: Accept-Encoding
< Content-Encoding: gzip
<
{ [data not shown]
100  596k    0  596k    0     0  5523k      0 --:--:-- --:--:-- --:--:-- 5574k
* Closing connection 0

real    0m0.112s
user    0m0.004s
sys     0m0.002s

without compression

admin@bastion1$ time curl -v -k -o/dev/null 'http://.../deps.js'
[...]
> GET .../deps.js HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: close
< Date: Wed, 09 Oct 2019 15:26:05 GMT
< Last-Modified: Mon, 02 Sep 2019 19:28:26 GMT
< ETag: "3bfdf549feec78af317650f9c35a7687"
< Content-Type: application/javascript
< Vary: Accept-Encoding
<
{ [data not shown]
100 1592k    0 1592k    0     0  52.2M      0 --:--:-- --:--:-- --:--:-- 53.6M
* Closing connection 0

real    0m0.034s
user    0m0.000s
sys     0m0.006s

haproxy compression and chunked encoding

0 Answers