How to read a 30G parquet file by python

user_16650130 0
Viewed 181

I am trying to read data from a large parquet file of 30G. My memory do not support default reading with fastparquet in python, so I do not know what I should do to lower the memory usage of the reading process.

fastparquet parquet python-3.x
1 Answers

You can use pyarrow's iter_batches to read back chunks of rows incrementally.

user_109943630
Related