Consider the following pseudo code:
pjs <- webdriver::run_phantomjs()
ses <- webdriver::Session$new(port = pjs$port)
ses$go("$randomurl$")
Question: will approach 1) and 2) yield the same amount of elements/nodes given an arbitrary $ANY_XPATH$?
Approach #1:
ses$findElements(xpath = $ANY_XPATH$)
Approach #2:
elem <- ses$findElement(xpath = "/*")
doc <- elem$getAttribute(name = "innerHTML") %>% xml2::read_html()
doc %>% rvest::html_nodes(xpath = $ANY_XPATH$)
Why do I ask? I realised that certain pages like the following do not fully render:
url <- "https://www.bs-krohn.de/#bewerber"
ses$go(url)
ses$takeScreenshot()
browseURL(url)
Obviously, this is more of an additional challenge, but it got me wondering if I can safely use approach 2. I would prefer approach 2, since it is faster than 1.
My best guess... would be that 1) and 2) should be equal and I could not falsify that hypothesis empirically so far.
My current understanding would be that phantomjs, as a headless browser, renders the page including all javascript snippets to be triggered. So
elem <- ses$findElement(xpath = "/*")
doc <- elem$getAttribute(name = "innerHTML") %>% xml2::read_html()
might be the equivalent of doing "right-click -> show source code" in a non-headless browser. But I am certainly not an expert for phantomjs and cant tell if there might be some lazy-loading(?) going on or something else I am not aware of.