(I edited the title, see EDIT below: GET path in the response contains spaces, requested path does not.)
I am working for a company that uses a "template system" to enable their markets (different countries) to host small web apps and making them "feel" like part of the main market website by pulling header, navigation, footer of the main website via curl:
So a php script uses curl to pull a special empty CMS page that contains only the relevant elements and placeholders for the web app to replace. This way, the web apps always have the current legal footer, header and navigation links to the main sections on the CMS and feel like being part of the cms albeit being hosted on their own somewhere else.
In this case, depending on an i18n parameter in the url, the script is curling a different template for each of ~50 markets.
Now the curious case: This is working for all templates but one, which is returning error 400 when being curled via https. But when curled via http, 301 is returned, then another request to the previously faulty https URL is made, only this time it returns 200.
Does not work:
* Trying [server_ip]...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562a5065a130)
* Connected to www.[company_name].pt ([server_ip]) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.[company_name].pt
* start date: Aug 23 19:02:44 2022 GMT
* expire date: Nov 21 19:02:43 2022 GMT
* subjectAltName: host "www.[company_name].pt" matched cert's "*.[company_name].pt"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562a5065a130)
> GET [path] HTTP/2
Host: www.[company_name].pt
User-Agent:Mozilla/5.0 (compatible; Integrator https://[app_name].[company_name].com)
Accept:text/html
Accept-Encoding:gzip,deflate
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive:115
Connection:keep-alive
Cache-Control:max-age=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* The requested URL returned error: 400
* stopped the pause stream!
* Connection #0 to host www.[company_name].pt left intact
Does work:
* Trying [server_ip]...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562a5066c0f0)
* Connected to www.[company_name].pt ([server_ip]) port 80 (#0)
> GET [path] HTTP/1.1
Host: www.[company_name].pt
User-Agent:Mozilla/5.0 (compatible; Integrator https://[app_name].[company_name].com)
Accept:text/html
Accept-Encoding:gzip,deflate
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive:115
Connection:keep-alive
Cache-Control:max-age=0
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html
< Content-Length: 185
< Connection: keep-alive
< Location: https://www.[company_name].pt/[path]
<
* Ignoring the response-body
* Connection #0 to host www.[company_name].pt left intact
* Issue another request to this URL: 'https://www.[company_name].pt/[path]'
* Trying [server_ip]...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562a5066c0f0)
* Connected to www.[company_name].pt ([server_ip]) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.[company_name].pt
* start date: Aug 23 19:02:44 2022 GMT
* expire date: Nov 21 19:02:43 2022 GMT
* subjectAltName: host "www.[company_name].pt" matched cert's "*.[company_name].pt"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562a5066c0f0)
> GET [path]
Host: www.[company_name].pt
User-Agent:Mozilla/5.0 (compatible; Integrator https://[app_name].[company_name].com)
Accept:text/html
Accept-Encoding:gzip,deflate
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive:115
Connection:keep-alive
Cache-Control:max-age=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< content-type: text/html; charset=utf-8
< content-length: 15305
< date: Thu, 22 Sep 2022 09:25:35 GMT
< server: Apache
< x-robots-tag: noindex
< strict-transport-security: max-age=31536000
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< referrer-policy: strict-origin-when-cross-origin
< cache-control: public, max-age=3600
< expires: Thu, 22 Sep 2022 10:25:35 GMT
< pragma: public
< accept-ranges: bytes
< content-encoding: gzip
<
* Connection #1 to host www.[company_name].pt left intact
Calling the template in the browser works, with http and https. curl from CLI works also for both http and https and returns 200, the error only occurs with PHP.
The script (a bit blown up for error search):
function curl($url){
$headers[] = "User-Agent:Mozilla/5.0 (compatible; Integrator https://[app_name].[company_name].com)";
$headers[] = "Accept:text/html";
$headers[] = "Accept-Encoding:gzip,deflate";
$headers[] = "Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$headers[] = "Keep-Alive:115";
$headers[] = "Connection:keep-alive";
$headers[] = "Cache-Control:max-age=0";
/**/debug_to_console("init curl");
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_ENCODING, "gzip");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSLVERSION, 6);
curl_setopt($curl, CURLOPT_FAILONERROR, 1);
curl_setopt($curl, CURLOPT_VERBOSE, 1);
curl_setopt($curl, CURLOPT_STDERR, $verbose = fopen('php://temp', 'rw+'));
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$data = curl_exec($curl);
if(curl_errno($curl)){
debug_to_console("curl url: " . $url);
debug_to_console("curl error: " . curl_error($curl));
//echo 'Request Error:' . curl_error($curl);
}
echo "Verbose information:\n<pre>", !rewind($verbose), htmlspecialchars(stream_get_contents($verbose)), "</pre>\n";
curl_close($curl);
return $data;
}
As mentioned, on other templates from other markets (on the same server) this is working fine. And I was told, that the instances are duplicated, so they all should have the same settings. Again, they are all on the same server. Unfortunately, I can't get them to check further until I'm pointing them into the right direction what to look at.
Tests with ssllabs.com showed no differences between the instances.
Being a Front End Dev, I'm really not getting any further at this point, after trying multiple suggestions for similar errors. I'm suspecting a wrong setting on that instance, but what instructions/hints can I give the server team?
Sorry I had to remove IPs and company/app names. Thank you.
Edit: Not sure if this is relevant, for some reason, the line > GET /[path] HTTP/2 is different in the faulty curl attempt:
Every successful curl has one space between the path and HTTP/2, the faulty one has three spaces between the path and HTTP/2.
When using http instead of https, the line also has three spaces > GET /[path] HTTP/1.1), gets the 301 response and after redirect, the new attempt has only one space > GET /[path] HTTP/2.
The URL is not the problem, it is absolutely identical to other instances where there is no problem. Does the server add a space character itself on the initial curl attempt? What could cause added spaces? This has to be server side, the other instances don't mangle the initial GET path.