Currently running some code like this:
df = pd.read_parquet('/tmp/my-file.parquet', engine='pyarrow')
I was having memory consumption issues since the files are large so I wanted to investigate is fastparquet would work better for memory usage.
When I switch the engine:
df = pd.read_parquet('/tmp/my-file.parquet', engine='fastparquet')
This line now throws an error like this:
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
I believe this is happening because I have an integer field that contains nulls. I can't find any documentation that says this is not supported.
Any ideas on why this is happening or how to workaround while still using fastparquet?