How to read a (.RDS) file from a URL in python?

Admin 1
Viewed 917

So I am trying to get data from NFLfastR and my R equivalent code is:

data <- readRDS(url('https://raw.githubusercontent.com/guga31bb/nflfastR-data/master/data/play_by_play_2019.rds'))
data

I have previously tried pyreadr module as well but that did not work for me. Currently I am using rpy2 module to make it work. Here is the code I am trying:

import rpy2.robjects as robjects
from rpy2.robjects import pandas2ri
import os
os.environ["R_HOME"] = r"C:\Program Files\R\R-3.6.3"
os.environ["PATH"]   = r"C:\Program Files\R\R-3.6.3\bin\x64" + ";" + os.environ["PATH"]

pandas2ri.activate()

readRDS = robjects.r['readRDS']
df = readRDS(url('https://raw.githubusercontent.com/guga31bb/nflfastR-data/master/data/play_by_play_2019.rds'))
df = pandas2ri.ri2py(df)
r python
3 Answers

Rds and Rdata files are difficult to read in other languages than R as the format although open is undocumented. Therefore there are not many options on how to read them in python. One is what you propose. Another is to use pyreadr, but you have to download the file to disk first as pyreadr cannot read directly from an url:

import pyreadr
from urllib.request import urlopen
link="https://raw.githubusercontent.com/guga31bb/nflfastR-data/master/data/play_by_play_2019.rds"
response = urlopen(link)
content = response.read()
fhandle = open( 'play_by_play_2019.rds', 'wb')
fhandle.write(content)
fhandle.close()
result = pyreadr.read_r("play_by_play_2019.rds")
print(result.keys())

EDIT

pyreadr 0.3.7 includes now a function to download files:

import pyreadr

url = "https://github.com/hadley/nycflights13/blob/master/data/airlines.rda?raw=true"
dst_path = "/some/path/on/disk/airlines.rda"
res = pyreadr.read_r(pyreadr.download_file(url, dst_path), dst_path)
user_95290480

In R, unlike Python, you do not have to qualify every function with its package source unless you face name conflicts. Additionally in R, there is no built-in method. Every function you call resides in a package. But R ships with default packages such as utils, base, stats for routine methods.

Specifically, your working R code calls two functions from base package as shown with double colon aliases:

nfl_url <- 'https://raw.githubusercontent.com/guga31bb/nflfastR-data/master/data/play_by_play_2019.rds'
data <- base::readRDS(base::url(NFL))

data

Consequently, you need to run the analogous procedure in Python's rpy2 by explicitly importing base package:

from rpy2.robjects.packages import importr
from rpy2.robjects import pandas2ri

base = importr("base")

nfl_url <- 'https://raw.githubusercontent.com/guga31bb/nflfastR-data/master/data/play_by_play_2019.rds'
r_df <- base.readRDS(base.url(nfl_url))

pandas2ri.activate()
py_df = pandas2ri.ri2py(r_df)
user_14224510

Well if you just want to read nflFastR data, you can directly read it in python as follows:

import pandas as pd
pd.read_csv('https://github.com/guga31bb/nflfastR-data/blob/master/data/' \
                         'play_by_play_2019.csv.gz?raw=True',
                         compression='gzip', low_memory=False)

But as of now, there is not a way to do this via python. Its hard enough to read an on-premises (.rds) file while reading from a url is something I never saw implemented. So you have to download the file on-premises then you can have read it directly using pyreadr package or rpy2 (if you have R installed) as you mentioned.

user_72129290
Related