I have staged a parquet file in internal snowflake user stage. When I try to use copy into command to load the data into snowflake table i am getting following error
Error parsing the parquet file: number of row groups exceeds the max limit (32767)
My parquet file contains roughly 9 million rows with 11 columns. Can any one guide me on how to overcome this issue.
Note : loading smaller parquet file with less rows works perfectly. problem occurs only when data is huge.
This is currently a limitation in Snowflake and there is a plan to address it in the near future. For the time being, you will need to split the files with fewer row groups per file.
You can use as example a Python lib to do that, see Dask and a simple example code:
# Load the required Library
import dask.dataframe as dd
# Read the source parquet file named "my_file.parquet"
ddf = dd.read_parquet('my_file.parquet')
# Split the source file into 3 files which will be stored in a sub-folder called "my_files"
# The resulting files found in this sub-folder are the files to ingest within Snowflake
ddf.repartition(3).to_parquet('my_files/')