Is there a way of reading shapefiles directly into R from an online source?

user_11430778 0
Viewed 96

I am trying to find a way of loading shapefiles (.shp) from an online repository/folder/url directly into my global environment in R, for the purpose of making plots in ggplot2 using geom_sf. In the first instance I'm using my Google Drive to store these files but I'd ideally like to find a solution that works with any folder with a valid url and appropriate access rights.

So far I have tried a few options, the first 2 involving zipping the source folder on Google Drive where the shapefiles are stored and then downloading and unzipping in some way. Have included reproducable examples using a small test shapefile:

  1. Using utils::download.file() to retrieve the compressed folder and unzipping using either base::system('unzip..') or zip::unzip() (loosely following this thread: Downloading County Shapefile from ONS):
# Create destination data folder (if there isn't one)
if(!dir.exists('data')) dir.create('data')
# Download the zipped file/folder 
download.file("https://drive.google.com/file/d/1BYTCT_VL8EummlAsH1xWCd5rC4bZHDMh/view?usp=sharing", destfile = "data/test_shp.zip")
# Unzip folder using unzip (fails)
unzip(zipfile = "data/test_shp.zip", exdir = "data/test_shp", junkpaths = TRUE)
# Unzip folder using system (also fails)
system("unzip data/test_shp.zip")

If you can't run the above code then FYI the 2 error messages are:

Warning message: In unzip(zipfile = "data/test_shp.zip", exdir = "data/test_shp", : error 1 in extracting from zip file

AND

End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of data/test_shp.zip or data/test_shp.zip.zip, and cannot find data/test_shp.zip.ZIP, period.

Worth noting here that I can't even manually unzip this folder outside R so I think there's something going wrong with the download.file() step.

  1. Using the googledrive package:
# Create destination data folder (if there isn't one)
if(!dir.exists('data')) dir.create('data')
# Specify googledrive url:
test_shp = drive_get(as_id("https://drive.google.com/file/d/1BYTCT_VL8EummlAsH1xWCd5rC4bZHDMh/view?usp=sharing"))
# Download zipped folder
drive_download(test_shp, path = "data/test_shp.zip")
# Unzip folder
zip::unzip(zipfile = "data/test_shp.zip", exdir = "data/test_shp", junkpaths = TRUE)
# Load test.shp
test_shp <- read_sf("data/test_shp/test.shp")

And that works!

...Except it's still a hacky workaround, which requires me to zip, download, unzip and then use a separate function (such as sf::read_sf or st_read) to read in the data into my global environment. And, as it's using the googledrive package it's only going to work for files stored in this system (not OneDrive, DropBox and other urls).

  1. I've also tried sf::read_sf, st_read and fastshp::read.shp directly on the folder url but those approaches all fail as one might expect.

So, my question: is there a workflow for reading shapefiles stored online directly into R or should I stop looking? If there is not, but there is a way of expanding my above solution (2) beyond googledrive, I'd appreciate any tips on that too!

Note: I should also add that I have deliberately ignored any option requiring the package rgdal due to its imminient permanent retirement and so am looking for options that are at least somewhat future-proof (I understand all packages drop off the map at some point). Thanks in advance!

sf google-drive-api shapefile r download
1 Answers

I ran into a similar problem recently, having to read in shapefiles directly from Dropbox into R.

As a result, this solution only applies for the case of Dropbox.

The first thing you will need to do is create a refreshable token for Dropbox using rdrop2, given recent changes from Dropbox that limit single token use to 4 hours. You can follow this SO post.

Once you have set up your refreshable token, identify all the files in your spatial data folder on Dropbox using:

shp_files_on_db<- drop_dir("Dropbox path/to your/spatial data/", dtoken = refreshable_token) %>% 
  filter(str_detect(name, "adm2"))

My 'spatial data' folder contained two sets of shapefiles – adm1 and adm 2. I used the above code to choose only those associated with adm2.

Then create a vector of the names of the shp, csv, shx, dbf, cpg files in the 'spatial data' folder, as follows:

shp_filenames<- shp_files_on_db$name

I choose to read in shapefiles into a temporary directory, avoiding the need to have to store the files on my disk – also useful in a Shiny implementation. I create this temporary directory as follows:

# create a new directory under tempdir
dir.create(dir1 <- file.path(tempdir(), "testdir"))

#If needed later on, you can delete this temporary directory 
unlink(dir1, recursive = T)

#And test that it no longer exists
dir.exists(dir1)

Now download the Dropbox files to this temporary directory:

for (i in 1: length(shp_filenames)){
  
  drop_download(paste0("Dropbox path/to your/spatial data/",shp_filenames[i]),
                dtoken = refreshable_token,
                local_path = dir1)
  
}

And finally, read in your shapefile as follows:

#path to the shapefile in the temporary directory
path1_shp<- paste0(dir1, "/myfile_adm2.shp")

#reading in the shapefile using the sf package - a recommended replacement for rgdal
shp1a <- st_read(path1_shp)
user_139041240
Related