Saving Intermediate Results of a Loop In Case of "Crash"?

antonoyaro8 0
Viewed 29

I have this loop in R:

output = list()

for (i in 1:999) 

{tryCatch({

{
  
    link_i<-  paste0(www.some_website, i+1,  /some_extension/, i,  .com)
    
    material_i <-  fromJSON(link_i)


    
   
    
   output[[i]] <- material_i

}
    
   }, error = function(e){})

}

Currently, if this Loop "crashes" - for example, if this loop crashes at the 998th iteration, I lose all my progress.

I have used the "tryCatch" statement to skip any errors that might be encountered while the loop is running. But I am interested in the following:

  • Suppose I click the "red stop sign button" in the corner of my screen and interrupt my loop - is there some code I can add to this loop that result in the "results" being saved up until the point that the loop was interrupted?

  • Suppose my computer shuts off (e.g. runs out of battery) - is there some code that I can add to this loop that would result in my work being saved as an "RDS" file on the computer? For example, after every 5 minutes, the intermediate work gets saved to "my documents"?

Thank you!

save loops r
1 Answers

It may be that you really need to use a loop; however, in most cases (not all), you could use a functional programming approach when you want to write a loop. The {purrr} package does that amazingly well and it also has a function called possibly(), which can be very helpful in your situation.

Assume I have a list of 5 numbers. I would like to multiply each number by 10 then take the square root of each value. The problem is that one of the 5 numbers was mistakenly input as a character rather than a numeric value (i.e "20" instead of 20). This means that an error will be produced when calculating the square root.

In the code below, I create a function for transforming the values: transform_values(). Then I apply the function to each element of the list. Notice that there is an error as expected.

# Load packages ----

library(purrr)


# List of values ----

l <- list(10, "20", 30, 40, 50)


# Create function ----

transform_values <- function(x){
  sqrt(x * 10)
}

# Apply function to each value of the list ----

map(l, transform_values)

Error in x * 10 : non-numeric argument to binary operator

Now I use possibly() function from purrr to create an "error safe" version of transform_values() and I call it possibly_transform_values(). This function returns NA (or anything else I want) if an error occurs and continues its process:


# Create error safe version of transform_values()

possibly_transform_values <- possibly(.f = transform_values, otherwise = NA)

# Apply it to each value of the list ----

map(l, possibly_transform_values)

[[1]]
[1] 10

[[2]]
[1] NA

[[3]]
[1] 17.32051

[[4]]
[1] 20

[[5]]
[1] 22.36068
SavedByJESUS0
Related