How to read parquet file from AWS S3 bucket using R without downloading it locally?

Question

I'm able to download the parquet file from AWS S3 bucket on local and then read from it (see the below code). But is there any way that I can directly read parquet file from S3 and read, without storing in local?

save_object("Financial_Sample.parquet", file = "Financial_Sample.parquet", bucket = 'my-bucket')
df <- read_parquet("Financial_Sample.parquet")```

user_12360444 · Answer

Take a look at the arrow package: https://arrow.apache.org/docs/r/index.html

It can directly read from s3 and even filter before reading using some dplyr verbs.

How to read parquet file from AWS S3 bucket using R without downloading it locally?

1 Answers