This is my first time posting so forgive me if I've missed including something. I'm fairly new to working with spatial data in R, let alone this volume of it, so any help would be appreciated! I'm trying to analyse an extremely large spatial dataset in R, and have made some good progress but am looking for a way to automate or loop my existing code so I don't need to go down my list of dataframes row by row. I'm a complete novice when it comes to loops and have no idea how to run multiple functions within one, if that's even possible. From what I've seen, folks also suggest writing custom functions and using dplyr but again - I'm pretty new to this whole thing!
I have almost 3000 shapefiles that are organised by their UK OS Grid Reference, and each shapefile has several thousand polygons - all in all totalling several hundred GB (more than I can read in in one go, at least and even if I could that would be pretty unwieldy).
In this code - for each UK county in a list, I'm matching its OS Grid reference against a list of shapefiles named by their OS Grid reference, and creating a vector of the matching ones. Using the vector, I use read_sf to read in all the corresponding shapefiles into a dataframe, which I then use rbind on. Once I have one long list of every polygon I run st_intersection on it to make sure everything is within the boundary of the chosen county. Once I have this intersected dataframe, I do some calculations about area and coverage (how much of the county is covered by polygons, etc.) and would like to add these area/coverage calculations as a new column to the original county dataframe. The issue there is, if I'm running through this list of counties one at a time, I'm not sure how to add the values to this new column one at a time as I go through the list.
gc()
library(sf)
library(sp)
library(ggplot2)
library(dplyr)
library(stringr)
#create dataframe of ~3000 shapefiles, organised by their UK OS Grid (1km) number
all_shps <- data_frame(filename = list.files(path = "Polygons", pattern = "\\.shp$", recursive = T))
all_shps <- all_shps %>%
mutate(filepath = paste0("Polygons/", filename))
#import additional layers
os_grid <- st_read("OSGB_Grid_1km.shp") #os grid (downloaded from https://github.com/charlesroper/OSGB_Grids)
county <- st_read("counties.shp") #https://www.data.gov.uk/dataset/11302ddc-65bc-4a8f-96a9-af5c456e442c/counties-and-unitary-authorities-december-2016-full-clipped-boundaries-in-england-and-wales
county_os <- st_intersection(os_grid, county) #adds OS tile ID to counties -> will be able to associate with x number of the million polygons
#i'd like to loop from here for(i in county){ ...?
countyx <- county$NAME[[1]] #work way down list of counties by row number
countyname <- county %>% filter(NAME == countyx) #preserve geometry
target <- county_os %>%
filter(NAME == countyx)
print(target$PLAN_NO)
targetgrid<-target$PLAN_NO #vector of relevant grid ids for county
shapes <- shapes_all[shapes_all$filename %in% target, ]#subset dataframe of filepaths listed in vector for chosen county
shapefile_list <- lapply(shapes$filepath, read_sf)
shapefile_list_all <- do.call(rbind, shapefile_list)
intersect_1 <- st_intersection(shapefile_list_all, countyname) #intersect polygons & chosen county
coverarea=sum(st_area(intersect_1))
countyarea=st_area(countyname)
prop_cover <- as.numeric(coverarea/countyarea) #i'd like to add these values as a new column to the original county dataframe
#or could potentially add these values to a vector as I run the loop, and then after its done add the vector to the original table with cbind?
So - is it possible to run these functions in a loop? Is there a more efficient way? Thank you for reading :)