I'm sending data back and forth to a PHP application using content-encoding: chunked through POSTs. I need my PHP application to read some data, work on it, send back a response, read some more data, and so on. I cannot read all data at once as it won't be available. Imagine a large file upload with a checksum being sent as a response at regular intervals.
The problem is that while I can read a handful of bytes from php://input, subsequent calls to fread do not return the new content.
At the moment I'm using PHP's Docker container. I tried both php:7.0-apache and php:5-apache with the same result.
The PoC client below generates random strings and sends them as chunks to the server at 3-second intervals. The server reads from php://input at 1-second intervals and prints the content. The server output shows only the first three strings are read; also the server seems to 'block' until the first three are read.
Things I've tried, to no avail:
- Using
fseek - Using
stream_selectdoes not seem to work with, er, thephp://inputstream. I have no idea why as this would be ideal for me, but given how poorly PHP is designed and implemented I'm not surprised. - Closing and re-opening
php://input - Using
fgetc
Client output:
$ python poc.py
Sending:
---
POST /poc.php HTTP/1.1
Host: localhost
accept-encoding: *;q=0
Transfer-Encoding: chunked
Content-Type: application/octet-stream
---
After sending headers, response:
HTTP/1.1 200 OK
Date: Mon, 29 May 2017 14:25:52 GMT
Server: Apache/2.4.10 (Debian)
X-Powered-By: PHP/5.6.30
transfer-encoding: chunked
Content-Type: application/octet-stream
4
OK
Waiting 3 seconds
Sending string: AuVuvsyGJc
Waiting 3 seconds
Sending string: LfKouYzccV
Waiting 3 seconds
Sending string: WmpPspYqiR
Waiting 3 seconds
Sending string: IApMOjoaIv
Waiting 3 seconds
Sending string: tuGrVklcVy
Waiting 3 seconds
Sending string: btUVIezCow
Waiting 3 seconds
Sending string: XUPOrEidyd
Traceback (most recent call last):
File "poc.py", line 33, in <module>
websock.send(to_chunk(rnd))
socket.error: [Errno 32] Broken pipe
Server output:
Connected
Read: AuVuvsyGJc
LfKouYzccV
WmpPspYqiR
Read:
Read:
Read:
Read:
172.17.0.1 - - [29/May/2017:14:25:52 +0000] "POST /poc.php HTTP/1.1" 200 191 "-" "-"
PHP server:
<?php
header("transfer-encoding: chunked");
header("content-type: application/octet-stream");
flush();
/**
* Useful to print debug messages in the Apache logs
*/
function _log($what) {
file_put_contents("php://stderr", print_r($what, true) . "\n");
}
_log("Connected");
/**
* To send data as chunks
*/
function _ch($chunk) {
echo sprintf("%x\r\n", strlen($chunk));
echo $chunk;
echo "\r\n";
flush();
}
// Test chunks
_ch("OK\r\n");
$web_php_input = fopen("php://input", 'r');
$continue = 5;
while ($continue-- > 0) {
$contents = fread($web_php_input, 1024);
_log("Read: " . $contents);
sleep(1);
}
fclose($web_php_input);
?>
Python client:
from __future__ import print_function
import random
import socket
import string
import time
def to_chunk(what):
return format(len(what), 'X') + "\r\n" + what + "\r\n"
websock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
websock.connect(("localhost", 8080))
# Send the initial chunked POST header
connect_string = ''.join((
"POST /poc.php HTTP/1.1\r\n",
"Host: localhost\r\n",
"accept-encoding: *;q=0\r\n", # ,gzip;q=0,deflate;q=0\r\n",
"Transfer-Encoding: chunked\r\n",
"Content-Type: application/octet-stream\r\n",
# "Connection: keep-alive\r\n",
"\r\n",
))
print("Sending:\n---\n{}\n---\n".format(connect_string))
websock.sendall(connect_string)
print("After sending headers, response:\n {}".format(websock.recv(1024)))
c = True
while c:
print("Waiting 3 seconds")
time.sleep(3)
rnd = ''.join(random.choice(string.ascii_letters) for _ in range(10))
rnd += '\r\n'
print("Sending string: {}".format(rnd))
websock.send(to_chunk(rnd))
print("done")
Dockerfile:
FROM php:5-apache
COPY custom.ini /usr/local/etc/php/conf.d
Docker command line:
docker build -t listener .
docker run -i --rm -p 8080:80 -v $(pwd):/var/www/html --name listener listener
custom.ini file to let PHP know that POST body should not be buffered:
enable_post_data_reading=false
Before someone else suggests using another language, or framework, or doing things differently: it has to be PHP; it cannot rely on any third-party library or PECL; and this is precisely what I need.
As a side note, this behaviour is compliant with the HTTP spec; a server does not have to read all inbound data before returning part of a response to a client. See also RFC6202.