Increase chunk size of Pyarrow FlightStreamReader

Viewed 133

Is there a way to increase the chunk size of the reader.read_chunk() from pyarrow flight? Where the reader is a pyarrow._flight.FlightStreamReader object from

reader = client.do_get(flight_info.endpoints[0].ticket, options)

Entire example.py script came from https://github.com/dremio-hub/arrow-flight-client-examples/blob/main/python/example.py

Currently each chunk is just 3968 rows. Was wondering if I can increase this chunk size to potentially increase throughput from downloading the results through the network and also minimize the number of times the read_chunk() is called (especially when the total number of rows is quite large)

0 Answers
Related