I have had my first go at using RSelenium today to scrape data from websites. I can navigate to the data I require via the tabs and drop-down menus (the hard bit?) but am now stuck at the point of extracting the actual data I need (the easy bit!)
My code so far is:
library(RSelenium)
checkForServer()
startServer()
remDr <- remoteDriver$new()
remDr$open()
remDr$navigate("https://www.whoscored.com/Teams/31")
webElem1 <- remDr$findElement(value = '//a[@href = "#team-squad-stats-detailed"]')
webElem1$clickElement()
webElem2 <- remDr$findElement("id", "category")
webElem2$clickElement()
webElem2$sendKeysToElement(list(key="down_arrow", key="down_arrow", key="down_arrow",
key="down_arrow", key="down_arrow", key="enter"))
webElem3 <- remDr$findElement("id", "subcategory")
webElem3$clickElement()
webElem3$sendKeysToElement(list(key="down_arrow", key="enter"))
webElem4 <- remDr$findElement("id", "statsAccumulationType")
webElem4$clickElement()
webElem4$sendKeysToElement(list(key="down_arrow", key="down_arrow", key="down_arrow",
key="enter"))
webElem5 <- remDr$findElement("id", "player-table-statistics-body")
Can someone advise the simplest way to now extract the data in this player table into csv form please? I am used to using the XML package and readHTMLTable to scrape other (static) websites but I am stuck on how to combine this with my RSelenium steps above.
Thank you
EDIT - having come back to this with fresh eyes the answer I have found is below:
webElem5 <- remDr$findElement(using = "id", value = "statistics-table-detailed")
webElem5txt <- webElem5$getElementAttribute("outerHTML")[[1]]
table <- readHTMLTable(webElem5txt, header=TRUE, as.data.frame=TRUE)[[1]]
This allows me to proceed with what I need on this part of the website.
If I may, I would like to ask for help with another part of the same site. I navigate to the data I need as follows:
remDr$navigate("https://www.whoscored.com/Matches/959894")
webElem1 <- remDr$findElement(using = "link text", value = "Match Centre")
webElem1$clickElement()
webElem2 <- remDr$findElement(value = '//a[@href = "#chalkboard"]')
webElem2$clickElement()
The data I would like to extract is in these boxes, but as the HTML doesn't say they are built as tables I don't really know how to proceed.
