Excel Selenium VBA - Web Scrape - Capture Data Points from Table

Viewed 31

I am trying to collate data from a website to Excel using VBA and Selenium. There are multiple product pages I need to click through, and for each product, I need to capture 8 data elements from a table that appears on each webpage.

The webpage table has two columns:

| Metric Name | Result |

| Strategist Style | None |

In my below code, I search for the metric name (e.g.: Strategist Style) and then return the text value beside it (i.e.: None) in the "result" column.

This loop works for 1 row only. I feel like I'm missing something obvious and could use some help.

The macro will continue to access new portfolio webpages and proceed through the loop--but only the first record will have its text copy/pasted onto the Excel spreadsheet. All other portfolios have blanks added to the spreadsheets instead of text from the webpage table.

I tested the code starting on different product webpages and it always works albeit just for the first record. Any help is appreciated. Thank you.


Dim bot As New WebDriver

'...

N = Worksheets("Scrape").Range("J:J").Cells.SpecialCells(xlCellTypeConstants).Count

'Set Do loop to stop when an empty cell is reached.
    Do Until IsEmpty(ActiveCell)
        
        'Clicks each portfolio profile page
        bot.FindElementByXPath("/html/body/div/div/div[5]/div[2]/div[2]/div/div[3]/div[3]/div[2]/div[1]/table/tbody/tr[" + CStr(N) + "]/td[1]/div").Click
        bot.Wait 3000

        'Copies information from Quick Facts table
        Strategist_Style = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Strategist Style')]//following::td[1]").Text
        ActiveCell.Offset(0, 1).Value = Strategist_Style
        
        Asset_Class = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Asset Class')]//following::td[1]").Text
        ActiveCell.Offset(0, 2).Value = Asset_Class
        
        Benchmark = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Benchmark')]//following::td[1]").Text
        ActiveCell.Offset(0, 3).Value = Benchmark

        Risk_Rating = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Risk Rating')]//following::td[1]").Text
        ActiveCell.Offset(0, 4).Value = Risk_Rating

        Risk_Score = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Risk Score')]//following::td[1]").Text
        ActiveCell.Offset(0, 5).Value = Risk_Score

        Account_Minimum = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Account Minimum')]//following::td[1]").Text
        ActiveCell.Offset(0, 6).Value = Account_Minimum
        
        Portfolio_Inception = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Portfolio Inception')]//following::td[1]").Text
        ActiveCell.Offset(0, 7).Value = Portfolio_Inception
        
        Avg_Annual_Turnover = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Avg. Annual Turnover')]//following::td[1]").Text
        ActiveCell.Offset(0, 8).Value = Avg_Annual_Turnover
        
        'Move to status column
        ActiveCell.Offset(0, 9).Select
        ActiveCell.Value = "Complete"
        ActiveCell.Offset(1, -9).Select
        
        'Sets counter one higher, returns to previous page
        N = N + 1
        bot.GoBack
        
    Loop

Update - 2022-09-19

I seemed to have solved this but I admittedly do not understand why it works. At the end of my VBA code, after I return to the previous page with bot.GoBack, I added a step to refresh the webpage (i.e.: bot.Refresh) and the loop successfully transcribes the 8 data elements for each product.

0 Answers
Related