Get RSelenium to print URL's that it's finished scraping

bandcar 0
Viewed 23

I'm running a loop to scrape a massive amount of data using RSelenium. If the loop breaks, I'd like to see the element and URL where RSelenium left off at.

Is there a way to print out the element the link is in and the url as each page is completed?

Using the below prints [[1]] [1] "" and that's it.

# check completed links
complete <- rd$findElement(using = "tag name", "a")
for(url in length(complete)){
     done <- complete[[url]]
     print(done$getElementText())
 }
rselenium for-loop r
1 Answers

You can use getCurrentUrl() instead of getElementText():

library(RSelenium)

driver <- rsDriver(browser = c("firefox"))
remote_driver <- driver[["client"]]

remote_driver$navigate("https://www.r-project.org/")
remote_driver$getCurrentUrl()

[[1]]
[1] "https://www.r-project.org/"
bretauv0
Related