I'm using Delphi 10 developing a project that automatically requests HTTPS to verify a third party service. Two days ago, IdHTTP started to return "403 Forbidden Error". So I installed Fiddler and added the fiddler's ssl certificate into Windows to decrypt SSL traffic. In IdHTTP I did:
HTTP.ProxyParams.ProxyServer := '127.0.0.1';
HTTP.ProxyParams.ProxyPort := 8888;
And like magic, the https service correctly answered 200 OK. So I removed the certificate and got 403 again, now observating the OnStatusInfo of IdSSLOpenSSL, which is:
SSL status: "before/connect initialization"
SSL status: "before/connect initialization"
SSL status: "SSLv2/v3 write client hello A"
SSL status: "SSLv3 read server hello A"
SSL status: "SSLv3 read server certificate A"
SSL status: "SSLv3 read server key exchange A"
SSL status: "SSLv3 read server done A"
SSL status: "SSLv3 write client key exchange A"
SSL status: "SSLv3 write change cipher spec A"
SSL status: "SSLv3 write finished A"
SSL status: "SSLv3 flush data"
SSL status: "SSLv3 read server session ticket A"
SSL status: "SSLv3 read finished A"
SSL status: "SSL negotiation finished successfully"
SSL status: "SSL negotiation finished successfully"
Cipher: name = ECDHE-ECDSA-AES128-GCM-SHA256; description = ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD
; bits = 128; version = TLSv1/SSLv3;
Doesn't look to have any TLS errors here. My code is this simple:
procedure TForm1.Button1Click(Sender: TObject);
var
http: TIdHTTP;
ssl: TIdSSLIOHandlerSocketOpenSSL;
HTMLSource: String;
begin
http:= TIdHTTP.Create(Nil);
ssl:= TIdSSLIOHandlerSocketOpenSSL.Create(Nil);
try
ssl.SSLOptions.Method:= sslvTLSv1_2;
ssl.SSLOptions.SSLVersions:= [sslvTLSv1_1, sslvTLSv1_2];
ssl.OnStatusInfo:= OnSSLStatusInfo;
http.IOHandler:= SSL;
//http.AllowCookies:= true;
//HTTP.ProxyParams.ProxyServer := '127.0.0.1';
//HTTP.ProxyParams.ProxyPort := 8888;
//http.Request.Accept:= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8';
//http.Request.UserAgent:= 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36';
//http.Request.Connection:= 'keep-alive';
http.HandleRedirects:= true;
http.ReadTimeout:= 15000;
HTMLSource:= http.Get('https://redacted.com/');
if (http.ResponseCode = 200) then
Memo1.Text:= HTMLSource;
finally
ssl.Free;
http.Free;
end;
All commented lines were tried as solutions, none of them worked. And with Fiddler certificate installed and proxy setted, I don't need cookies, user-agent, or anything else to work. Maybe the service somehow fingerprinted Indy automatic requests, and Fiddler's proxy spoof this? Still don't know how to fix the issue without using Fiddler proxy/certificate.
The logs for IdConnectionIntercept are:
GET / HTTP/1.1
Host: redacted.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: identity
User-Agent: Mozilla/3.0 (compatible; Indy Library)
---
HTTP/1.1 403 Forbidden
Server: cloudflare
Date: Thu, 11 Jun 2020 00:24:04 GMT
Content-Type: text/html
Content-Length: 151
Connection: keep-alive
CF-RAY: 5a172fe59902d07a-CWB
cf-request-id: 03425c437f0000d07a0696c200000001
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
The logs running with IdHTTP setted to Fiddler's proxy (and Fiddler certificate to decrypt SSL traffic are):
CONNECT redacted.com:443 HTTP/1.1
Pragma: no-cache
Proxy-Connection: keep-alive
Host: redacted.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/3.0 (compatible; Indy Library)
---
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 21:29:27.471
Connection: close
---
GET / HTTP/1.1
Host: redacted.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: identity
User-Agent: Mozilla/3.0 (compatible; Indy Library)
---
HTTP/1.1 301 Moved Permanently
Date: Thu, 11 Jun 2020 00:29:19 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d2453057d13003b17569dc77d925d4bd61591835359; expires=Sat, 11-Jul-20 00:29:19 GMT; path=/; domain=.redacted.com; HttpOnly; SameSite=Lax
X-Sorting-Hat-PodId: 80
Location: https://www.redacted.com/
X-Request-Id: df4074f2-a140-4488-9831-848a432641b8
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
CF-Cache-Status: DYNAMIC
cf-request-id: 0342610fb40000d082169fc200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5a173792bedcd082-CWB
alt-svc: h3-27=":443"; ma=86400
<html><body>You are being <a href="https://www.redacted.com/">redirected</a>.</body></html>
Some headers of the 301 answer were removed for privacy.
Somehow the certificate installed by Fiddler to decrypt SSL traffic is making the request work. I just don't have any theory about why, and how to fix this issue inside the code.