I was trying to save a pandas dataframe to parquet format using Pyarrow v2, and I ran into a weird problem. The (simplified) dataframe has one string column and one nested column (list of dicts). Here is an example:
labels = ["aa", 'bb', 'cc', 'dd']
vals = [random.choice(labels) for _ in range(2000)]
df = pd.DataFrame({'names': vals})
df["name_nested"] = df.names.apply(lambda x: [{"label": x}])
df.to_parquet("x.par")
as you see the columns "names" and "name_nested" should have the same values:
df.head()
names name_nested
0 bb [{'label': 'bb'}]
1 aa [{'label': 'aa'}]
2 cc [{'label': 'cc'}]
3 cc [{'label': 'cc'}]
4 cc [{'label': 'cc'}]
However, once I reload from disk the saved parquet file, things are weird, I don't get same result:
df2 = pd.read_parquet("x.par")
df2["name_nested2"] = df2.name_nested.apply(lambda x: x[0]["label"])
len(df2[df2.name_nested2 != df2.names])
# 726
Out of the 2000 entries, there are 726 entries which do not match. here is an example:
df2[df2.name_nested2 != df2.names]
names name_nested name_nested2
1025 dd [{'label': 'cc'}] cc
1027 bb [{'label': 'aa'}] aa
1029 aa [{'label': 'cc'}] cc
1031 dd [{'label': 'aa'}] aa
1035 bb [{'label': 'dd'}] dd
As you see the column name_nested is not the same as names anymore! This is very wrong behavior. also I noticed that this happens only if the dataframe has more than 1024 rows, and the mismatches happen only after row 1024 too.
I thought this would be a known issue, but couldn't find any information about this.
Once I upgraded to Pyarrow 6, this is no longer an issue, but wanted to understand the root cause of this, if anyone have seen this before.