How can I write a pandas dataframe to disk in .arrow format? I'd like to be able to read the arrow file into Arquero as demonstrated here.
How to write a pandas dataframe to .arrow file
Viewed 3500
2 Answers
Since Feather is the Arrow IPC format, you can probably just use write_feather. See http://arrow.apache.org/docs/python/feather.html
You can do this as follows:
import pyarrow
import pandas
df = pandas.read_parquet('your_file.parquet')
schema = pyarrow.Schema.from_pandas(df, preserve_index=False)
table = pyarrow.Table.from_pandas(df, preserve_index=False)
sink = "myfile.arrow"
# Note new_file creates a RecordBatchFileWriter
writer = pyarrow.ipc.new_file(sink, schema)
writer.write(table)
writer.close()
Related