Is there a pyarrow equivalent of the chunksize argument in pandas.read_csv?

user_8283047 0
Viewed 2774

I am looking to process a large file(5 gb) in RAM but am getting an out of memory error. Is there a way to process the parquet file in chunks like there is in pandas.read_csv?

import pyarrow.parquet as pq

    def main():
        df = pq.read_table('./data/train.parquet').to_pandas()            

    main()
parquet pyarrow pandas
1 Answers

There is not yet, but there are issues open about adding this option (see https://issues.apache.org/jira/browse/ARROW-3771, others). Note that memory use will be significantly improved in the upcoming 0.12 release.

In the meantime, you can use pyarrow.parquet.ParquetFile and its read_row_group method to read one row group at a time.

user_7765600
Related