We're trying to get Webdav running in Kubernetes using an Azure Files storage backend, which is mounted in the container on /dav/data. The container itself is running Alpine Linux 3.12.1, in which we're installing all our required Apache packages.
All is well and good when not mounting this storage, or when using a different storage backend. However, when mounting the Azure Files storage, stuff starts to break.
Uploading files works without issue, but downloading does not; most software complains about invalid HTTP headers/responses. When investigating this further, I see that the beginning of the headers seems to be getting cut off.
Example headers of a correct response (obtained by not mounting the volume):
HTTP/1.1 200 OK
Date: Tue, 01 Dec 2020 13:54:53 GMT
Server: Apache/2.4.46 (Unix)
Last-Modified: Tue, 01 Dec 2020 13:51:02 GMT
ETag: "bla"
Accept-Ranges: bytes
Content-Length: 985
Connection: close
Example headers of an incorrect response:
s: bytes
Content-Length: 985
Connection: close
Everything up to the first s in the Accept-Ranges header seems to be getting eaten somewhere. There also seem to be a number of extra null bytes at the end of the response.
In an effort to get to the bottom of this I looked into logging as much as I possibly could, and stumbled upon the DumpIO module, which would allow me to log both the response headers as well as the body. For some reason, loading this module, setting DumpIOOutput On and LogLevel dumpio:trace7 actually fixes the issue. Response headers are fine, and the response body is exactly what you'd expect. And it's driving me nuts.
I suspect there's some kind of weird buffer/window issue being caused by an interaction between Apache and the mounted volume, but I haven't been able to figure out what.
We've since changed the storage backend used for the volume, but I'd still really like to know what caused this issue.
I've also been able to reproduce this locally in Docker.