VBA web scraping: Difference between internet explorer and XMLHTTP request

Question

I am learning web scraping in VBA. And i am unable to understand the difference in results of requests sent through internet Explorer and XMLHTTP request.

This is the website -- http://www.scstrade.com/stockscreening/SS_CompanySnapShotQF.aspx?symbol=LUCK

Scraping through IE writes perfect results in the text file but XMLHTTP request writes empty string in the text file. I can not understand the reason. The code for each method is given below:

Dim ie As SHDocVw.InternetExplorer
Dim ele As MSHTML.IHTMLElement
Dim page As MSHTML.HTMLDocument
Dim fso As Scripting.FileSystemObject
Dim st As Scripting.TextStream
   
Set ie = New SHDocVw.InternetExplorer
ie.Visible = True
ie.navigate "http://www.scstrade.com/stockscreening/SS_CompanySnapShotQF.aspx?symbol=DGKC"

Do While ie.Busy Or ie.readyState <> READYSTATE_COMPLETE
Loop

Set ele = ie.document.getElementById("listisq1")

Set fso = New Scripting.FileSystemObject
Set st = fso.CreateTextFile(ThisWorkbook.Path & "\myFile.txt")
st.Write ele.innerText

And the code for XMLHTTP request is as follows

Dim page As MSHTML.HTMLDocument
Dim req As New MSXML2.XMLHTTP60
Dim ele As MSHTML.IHTMLElement
Dim fso As Scripting.FileSystemObject
Dim st As Scripting.TextStream

req.Open "GET", "http://www.scstrade.com/stockscreening/SS_CompanySnapShotQF.aspx?symbol=LUCK", False
req.send

If req.Status <> 200 Then
    MsgBox "req failed"
    Exit Sub
End If

Set page = New MSHTML.HTMLDocument

page.body.innerHTML = req.responseText

Set ele = page.getElementById("listisq1")

Set fso = New Scripting.FileSystemObject
Set st = fso.CreateTextFile(ThisWorkbook.Path & "\XMLHTTPscrape.txt")
st.Write ele.innerText

And this method (XMLHTTP) is writing nothing in the text file. Please guide me if there is something wrong with the code or is there an inherent difference in the two methods. Thanks

user_9189799 · Answer

This following should give you the content of this portion in json response.

Sub GetContent()
    Const link = "http://www.scstrade.com/stockscreening/SS_CompanySnapShotQF.aspx/isq4"
    Dim payload As Variant

    payload = "{'sym':'LUCK','_search':false,'nd':1600525542589,'rows':30,'page':1,'sidx':'Year / Quarter','sord':'desc'}"
    
    With CreateObject("MSXML2.XMLHTTP")
        .Open "POST", link, False
        .setRequestHeader "Content-Type", "application/json"
        .send payload
        MsgBox .responseText
    End With
End Sub

VBA web scraping: Difference between internet explorer and XMLHTTP request

1 Answers