My use case is I process ~100MB on a daily basis. I was using Pandas DataFrame as individual files, but this fails due to pandas's tendency to coerce dtypes dependent on the data from different days. I try to read these with Dask DataFrame and it fails due to the different schemas. With descriptive column names and 717 columns the exception message is unmanageable (100KB of dense binary strings in fixed lengths).
So I have attempted to use Dask to write out a mega parquet and hope it sorts out pandas dtype skullduggery. Sometimes I need to re-process a day or 2 in the middle of the full range of days data I already have.
I managed to come up with this so far, it is very ugly and I cannot help but think there is a better way. There appears to be no way of using filters with the read_parquet, because we filter on index. There does not seem to be a way of negating a range on the index values. The index is just a date, no hours etc. df is the day's worth of data and the mdf is my mega-df with a years data in it
mdf = dd.read_parquet(self.local_location + self.megafile, engine='pyarrow')
inx = df.index.unique()
start1 = '2016-01-01'
end1 = pd.to_datetime(inx.values.min()).strftime('%Y-%m-%d')
start2 = pd.to_datetime(inx.values.max()).strftime('%Y-%m-%d')
end2 = '2029-01-01'
mdf1 = mdf[start1:end1]
mdf2 = mdf[start2:end2]
if len(mdf1) > 0:
df_usage1 = 1 + mdf1.memory_usage(deep=True).sum().compute() // 100000001
if len(mdf2) > 0:
df_usage2 = 1 + mdf1.memory_usage(deep=True).sum().compute() // 100000001
mdf1 = mdf1.append(mdf2, npartitions=df_usage2)
else:
if len(mdf2) > 0:
df_usage2 = 1 + mdf2.memory_usage(deep=True).sum().compute() // 100000001
mdf1 = dd.from_pandas(df).append(mdf2, npartitions=df_usage2)
this also throws an exception at
mdf1 = mdf1.append(df, npartitions=df_usage1)
{ValueError}Exactly one of npartitions and chunksize must be specified.
Funny that, because that is exactly what I am doing.
df_usage2 in this case = 2
Alternative better approach sought and maybe an explanation on what it actually wrong with the append.