I have a list of url's in R. These are stored in a list object called "url".
I also have this loop here that I am using for webscraping. In this loop, if a specific url takes longer than 1 second to finish the webscraping, the loop automatically moves to the next url:
library(R.utils)
res <- vector("list", length = 10L)
for (i in 1:10) {
print(i)
tryCatch({
res[[i]] <- withTimeout({
url_i = url[i]
res_i <- data.frame(fromJSON(url_i))
res[[i]] <- r_i
print(i)
}, timeout = 1)
}, TimeoutException = function(ex) {
message("Timeout. Skipping.")
res[[i]] <- NULL
})
}
This loop runs, but the problem is that it seems like the loop is only returning the "indices" instead of the actual webscraping results. For example:
> res
[[1]]
[1] 1
[[2]]
[1] 2
[[3]]
[1] 3
Have I written this loop wrong? And what can I do to fix this?
Thank you!