How to crawl cellphone apps that use HTTP connect method with Fiddler?

user_17364708 0
Viewed 139

I'm new to web-crawlers, trying to crawl ridership data of metro from the cellphone maps app(www.amap.com) with Fiddler, but I got this HTTP connect method, which is not viewable. There are icons of locks next to the URL and in 'Response' it says this: 'Encrypted HTTPS traffic flows through this CONNECT tunnel. HTTPS Decryption is enabled in Fiddler, so decrypted sessions running in this tunnel will be shown in the Web Sessions list.'

Captured traffics of my cellphone

I found a solution suggesting that customizing rules in fiddler may help, so I followed and added this to its script:

if (oSession.oRequest[‘User- Agent’].IndexOf("Android") > -1 && oSession.HTTPMethodIs("connect")) {
    oSession.oResponse.headers["Connection"]="Keep-Alive";
}

The changes to Fiddler Script

But of course, it didn't work, I've tried both iPhone and android and changed the header in the script respectively, none of them helped.

So is this app and HTTP connect method crawlable? The data is constructively helpful to my research, instead, it is not provided in website 'amap', so it has to be done through a cellphone.

httpconnection fiddler web-crawler android
1 Answers

If you have HTTPS decryption enabled in Fiddler but you see (mostly) only CONNECT requests this means that the apps on the device try to open a connection but do not trust the Fiddler root certificate.

If you try to use the apps on-device you will notice that there is currently no working network connection available (requests just don't work as the apps don't accept the server certificate created by Fiddler).

On Android devices since Andorid 6 you need root permissions to instal the Fiddler rot certificate or alternatively if you want to monitor a single app you can try to modify and re-sign the app. All details are described in this question and answer: Some androids apps won't connect through fiddler

user_1509780
Related