How can I store data that needs to be accessed from multiple R Projects whilst retaining code portability?

user_4957167 0
Viewed 131

How can I store data that needs to be accessed from multiple R Projects whilst retaining code portability?

From within a project that I'm working on in R, I can access data stored approximately anywhere on my system. But if it is at an arbitrary location outside of the project then that hampers code portability.

I have projects that I want to be able to work on on two different machines, with different directory setups. For data that can be stored within the project, this can work fine as I can use tools like setting the project up as an RStudio Project and using the here package to refer to the data's location.

However, I have some data that (a) I want to use in multiple projects and (b) is pretty large. Consequently, I don't want to duplicate it to all of the projects I might use it in.

I can kind of work around this for projects where only I am working on them, by setting up a consistent folder structure at least relative to the projects and using relative paths. E.g., I can resolve to store these multi-project datasets in a Data folder that's two levels up from where the project lives.

shared_data_location <- "../../Data/"

But where I'm working on project with collaborators, even that is a bit of an imposition / assumption about where they will be able to put their code folders and data folders.

r project
1 Answers

A couple of ideas:

  • Use packages. I think a package would be fine if the data are not too large. Put on GitHub and then devtools::install_github("jamse/cool_data") and then library(cool_data) in R.

  • Use environment variables. So SHARED_DATA="/Users/jamse/shared/Data" could be put in an .Renviron file. Other users might need to choose a slightly different form. But then you could use Sys.getenv("SHARED_DATA") to access this in R code.

  • Use a database. This involves more overhead, but R works nicely with databases (especially dplyr). I have essentially no data files on my computers that I access directly. Everything is in PostgreSQL (as discussed by Hadley Wickam, PostgreSQL has more features than similar systems, but there is some setup cost involved). This pays off big time if you're using other languages (e.g., Python), which can generally access the data as well.

    Note that things can get complicated if sharing with other users and you have limited IT support. But potentially the payoff can be even greater (e.g., I share my database server with others, so data becomes even more accessible).

user_7457370
Related