I scraped a bunch of url's and was using RSelenium to add the links to the archive.org website. My loop was about half way finished when it broke. So I used the which() function to determine where it left off, then I restarted the loop from that point.
The problem is that the driver kept typing the wrong index instead of the one I specified in the loop. To fix the problem, I subset 'mget_links' to 'mget_links2' so that it only contained the remaining links I needed. I ran the loop again and it somehow still typed an index from 'mget_links'. The driver keeps typing in this link: https://www.bjjcompsystem.com/tournaments/1869/categories/2053146
What is going on here? How do I fix it?
library(rvest)
library(tidyverse)
library(RSelenium); library(netstat)
pageMen = read_html('https://www.bjjcompsystem.com/tournaments/1869/categories')
mget_links <- pageMen %>%
html_nodes('.categories-grid__category a') %>%
html_attr('href') %>%
paste0('https://www.bjjcompsystem.com', .)
# know the length for the loop
length(mget_links)
remote_driver = rsDriver(browser = 'firefox',
verbose = F,
port = free_port())
rd = remote_driver$client
rd$open()
rd$navigate('https://web.archive.org/save')
rd$maxWindowSize()
for (i in 1:length(mget_links[255:259])){
save_page_box = rd$findElement(using = 'xpath', '//*[(@id = "web-save-url-input")]')
save_page_box$clickElement()
save_page_box$sendKeysToElement(list(mget_links[i], key='enter'))
Sys.sleep(26)
return.to.save.page = rd$findElement(using = 'link text', 'Return to Save Page Now')
return.to.save.page$clickElement()
}
# Reran above code with mget_links2 instead of mget_links[255:259]. I also tried #mget_links2[1:156] with no luck.
mget_links2= mget_links[104:259]