I am trying to collate data from a website to Excel using VBA and Selenium. There are multiple product pages I need to click through, and for each product, I need to capture 8 data elements from a table that appears on each webpage.
The webpage table has two columns:
| Metric Name | Result |
| Strategist Style | None |
In my below code, I search for the metric name (e.g.: Strategist Style) and then return the text value beside it (i.e.: None) in the "result" column.
This loop works for 1 row only. I feel like I'm missing something obvious and could use some help.
The macro will continue to access new portfolio webpages and proceed through the loop--but only the first record will have its text copy/pasted onto the Excel spreadsheet. All other portfolios have blanks added to the spreadsheets instead of text from the webpage table.
I tested the code starting on different product webpages and it always works albeit just for the first record. Any help is appreciated. Thank you.
Dim bot As New WebDriver
'...
N = Worksheets("Scrape").Range("J:J").Cells.SpecialCells(xlCellTypeConstants).Count
'Set Do loop to stop when an empty cell is reached.
Do Until IsEmpty(ActiveCell)
'Clicks each portfolio profile page
bot.FindElementByXPath("/html/body/div/div/div[5]/div[2]/div[2]/div/div[3]/div[3]/div[2]/div[1]/table/tbody/tr[" + CStr(N) + "]/td[1]/div").Click
bot.Wait 3000
'Copies information from Quick Facts table
Strategist_Style = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Strategist Style')]//following::td[1]").Text
ActiveCell.Offset(0, 1).Value = Strategist_Style
Asset_Class = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Asset Class')]//following::td[1]").Text
ActiveCell.Offset(0, 2).Value = Asset_Class
Benchmark = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Benchmark')]//following::td[1]").Text
ActiveCell.Offset(0, 3).Value = Benchmark
Risk_Rating = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Risk Rating')]//following::td[1]").Text
ActiveCell.Offset(0, 4).Value = Risk_Rating
Risk_Score = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Risk Score')]//following::td[1]").Text
ActiveCell.Offset(0, 5).Value = Risk_Score
Account_Minimum = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Account Minimum')]//following::td[1]").Text
ActiveCell.Offset(0, 6).Value = Account_Minimum
Portfolio_Inception = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Portfolio Inception')]//following::td[1]").Text
ActiveCell.Offset(0, 7).Value = Portfolio_Inception
Avg_Annual_Turnover = bot.FindElementByXPath("//table[@class='databox']//tr/td[contains(.,'Avg. Annual Turnover')]//following::td[1]").Text
ActiveCell.Offset(0, 8).Value = Avg_Annual_Turnover
'Move to status column
ActiveCell.Offset(0, 9).Select
ActiveCell.Value = "Complete"
ActiveCell.Offset(1, -9).Select
'Sets counter one higher, returns to previous page
N = N + 1
bot.GoBack
Loop
Update - 2022-09-19
I seemed to have solved this but I admittedly do not understand why it works. At the end of my VBA code, after I return to the previous page with bot.GoBack, I added a step to refresh the webpage (i.e.: bot.Refresh) and the loop successfully transcribes the 8 data elements for each product.