How to compress parquet file with zstandard using pandas

user_11554863 0
Viewed 2295

i'm using pandas to convert dataframes to .parquet files using this command:

df.to_parquet(file_name, engine='pyarrow', compression='gzip')

I need to use zstandard as compression algorithm, but the function above accepts only gzip, snappy, and brotli. I tried Is there a way to include zstd in this function? If not, how can i do that with other packages? I tried with zstandard, but it seems to accept only bytes-like objects.

zstandard pandas dataframe python
2 Answers

I usually use zstandard as my compression algorithm for my dataframes.

This is the code I use (a bit simplified) to write those parquet files:

import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa

parquetFilename = "test.parquet"

df = pd.DataFrame(
    {
        "num_legs": [2, 4, 8, 0],
        "num_wings": [2, 0, 0, 0],
        "num_specimen_seen": [10, 2, 1, 8],
    },
    index=["falcon", "dog", "spider", "fish"],
)

df = pa.Table.from_pandas(df)
pq.write_table(df, parquetFilename, compression="zstd")

And to read these parquet files:

import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa

parquetFilename = "test.parquet"
df = pq.read_table(parquetFilename)
df = df.to_pandas()

For more details see these sites for more information:

Finally a shameless plug for a blog post I wrote. It is about the speed vs space balance of zstandard and snappy compression in parquet files using pyarrow. It is relevent to your question and includes some more "real world" code examples of reading and writing parquet files in zstandard. I will actually be writing a follow up soon too. if you're interested let me know.

user_83966840
user_1275080
Related