How can I read a 63 GB .csv file into RStudio from the Allen Brain Map using R?

user_14067718 0
Viewed 244

Using RStudio, I am trying to read in the Gene_expression_matrix.csv file from the Brain Allen Institute, and the file is too large, even for computers with large amounts of RAM (I have access to and have tried it on a laptop with 64 GB RAM and a computer with 384 GB RAM. Has anyone accessed this file or any of a similar size? Thanks!

I'm using this code:

Gene_expression_matrix <- read.csv("Gene_expression_matrix.csv")

The error message I receive is:

Error: cannot allocate vector of size 3.9 Mb
rna-seq bigdata r
2 Answers

You can use disk.frame like this

library(disk.frame)
setup_disk.frame()

Gene_expression_matrix.df <- csv_to_disk.frame(
   "Gene_expression_matrix.csv",
   outdir = "c:/this/is/where/the/output/is" # specify a path for where you want to save the file
)

If the above fails, then try to limit the amount you read by specifying in_chunk_size which will only read in_chunk_size rows at a time to limit RAM usage. E.g.

Gene_expression_matrix.df <- csv_to_disk.frame(
   "Gene_expression_matrix.csv",
   outdir = "c:/this/is/where/the/output/is", # specify a path for where you want to save the file
   in_chunk_size = 1e7 # read 10 million rows at a time; adjust down if still runs of out RAM
)

Once the data is loaded, you can use dplyr verbs and some common functions to look at your data. See this quick start.

For example

head(Gene_expression_matrix.df)

I am sure {disk.frame} can help in this case as it is designed for this! If you run into issues, please raise a ticket here and I will help you.

user_2399230

try this library

library('data.table')
Gene_expression_matrix <- fread("Gene_expression_matrix.csv")

it is extremely faster than read.csv.

user_81623940
Related