I'm building a scraper in Excel VBA to pull data from this link at McMasterCarr.com.
'Sub McMaster2()
Dim htmlTable As IHTMLElement
Dim htmlRow As IHTMLElement
Dim i As Integer
Dim ieObj As Object
Set ieObj = CreateObject("InternetExplorer.application")
ieObj.navigate "https://www.mcmaster.com/screws/" & _
"socket-head-screws/socket-head-screws-6/" & _
"high-strength-316-stainless-steel-socket-head-screws/"
Application.Wait Now + TimeValue("00:00:05")
For Each htmlTable In ieObj.document.getElementsByClassName("hn")
For Each body In htmlTable.getElementsByTagName("tbody")
Debug.Print body.childElementCount & " rows"
Next body
Next htmlTable
End Sub'
The immediate window shows 57 rows. When I navigate to the webpage and check the DOM, it lists 80 rows. DOM screenshot clip
On further review, it appears 57 rows is the most it will return. For instance, if you change the .navigate string to "https://www.mcmaster.com/screws/socket-head-screws/socket-head-screws-6/brass-socket-head-screws/", the immediate window reports 57 rows even though the DOM lists 123 children.
The upshot is I can only read in the first 57 rows of the table. What am I doing wrong? Is there something magic about 57?