parallelisation returns NULL - issue with terra?

user_3384265 0
Viewed 52

When parallelising (via %dopar%) the code as shown in create_routes2 it returns NULL. When not parallelising (create_routes1, via %do%), I get the result I want.

I've read that terra can have issues with being parallelised but I'm not directly using terra within the parallelisation, so I'm not sure why this is the case?

library(devtools)
install_github("josephlewis/leastcostpath@dev")
library(leastcostpath)
library(foreach)
library(doParallel)

r <- terra::rast(system.file("extdata/SICILY_1000m.tif", package="leastcostpath"))

slope_cs <- create_slope_cs(x = r, cost_function = "tobler", neighbours = 4)

locs <- sf::st_sf(geometry = sf::st_sfc(
 sf::st_point(c(839769, 4199443)),
 sf::st_point(c(1038608, 4100024)),
 sf::st_point(c(907695, 4145478)),
 sf::st_point(c(1054446, 4232288)),
 sf::st_point(c(957989, 4208863)),
 crs = terra::crs(r)))

# this function is called within the foreach
# View(leastcostpath::create_lcp)

create_routes1 <- function (x, locations, cost_distance = FALSE) 
{
 network <- expand.grid(1:nrow(locations), 1:nrow(locations))
 network <- network[network[, 1] != network[, 2], ]
 lcp_network <- foreach::foreach(i = 1:nrow(network), .packages = c("leastcostpath"), 
                                 .errorhandling = "remove", .combine = "rbind") %do% 
   {
     lcp <- (create_lcp(x = x, origin = locations[network[i, 1], ], destination = locations[network[i, 2],], cost_distance = cost_distance))
     lcp$origin_ID <- network[i, 1]
     lcp$destination_ID <- network[i, 2]
     return(lcp)
   }
 
 return(lcp_network)
}

lcps1 <- create_routes1(x = slope_cs, locations = locs, cost_distance = FALSE)

#####

create_routes2 <- function (x, locations, cost_distance = FALSE, ncores = 2)
{
 
 myCluster <- makeCluster(ncores)
 registerDoParallel(myCluster)
 
 network <- expand.grid(1:nrow(locations), 1:nrow(locations))
 network <- network[network[, 1] != network[, 2], ]
 lcp_network <- foreach::foreach(i = 1:nrow(network), .packages = c("leastcostpath"), 
                                 .errorhandling = "remove", .combine = "rbind") %dopar% 
   {
     lcp <- (create_lcp(x = x, origin = locations[network[i, 1], ], destination = locations[network[i, 2],], cost_distance = cost_distance))
     lcp$origin_ID <- network[i, 1]
     lcp$destination_ID <- network[i, 2]
     return(lcp)
   }
 
 stopCluster(myCluster)
 
 return(lcp_network)
}

lcps2 <- create_routes2(x = slope_cs, locations = locs, cost_distance = FALSE, ncores = 2)
terra sf parallel-processing r foreach
2 Answers

Most likely because your x = sc_slope object and/or your locations = locs object cannot be exported to another R process (here, a parallel worker). See https://future.futureverse.org/articles/future-4-non-exportable-objects.html#package-terra for more and other examples of this problem. Notice also the mentioning of wrap() and vect() for a workaround.

PS. It does not matter which parallelization framework you use in R; this is a problem that happens whenever you try to send objects to another R process running in the background.

user_10720910
user_33842650
Related