I'm testing the apply() method of Dask dataframes and I'm running this code:
import pandas as pd
import dask.dataframe as dd
import time
def enrich_str(str):
val1 = f'{str}_1'
val2 = f'{str}_2'
val3 = f'{str}_3'
time.sleep(3)
return val1, val2, val3
def enrich_row(passed_row):
col_name = str(passed_row['colName'])
my_string = str(passed_row[col_name])
val1, val2, val3 = enrich_str(my_string)
passed_row['enriched1'] = val1
passed_row['enriched2'] = val2
passed_row['enriched3'] = val3
return passed_row
df = pd.DataFrame({'numbers': [1, 2, 3, 4, 5], 'colors': ['red', 'white', 'blue', 'orange', 'red']},
columns=['numbers', 'colors'])
ddf = dd.from_pandas(df, npartitions=2)
ddf['colName'] = 'colors'
result = ddf.apply(enrich_row, axis=1,
meta={'numbers': 'int64', 'colors': 'string', 'colName': 'string',
'enriched1': 'string', 'enriched2': 'string', 'enriched3': 'string'})
tic = time.perf_counter()
enriched_df = result.compute()
toc = time.perf_counter()
print(f"{enriched_df.shape[0]} rows enriched in {toc - tic:0.4f} seconds")
print(enriched_df)
The final result is correct, but I'm getting these warnings:
5 rows enriched in 9.0715 seconds :17: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched1'] = val1 C:\Users\LZavarella\miniconda3\envs\pbi_powerquery_env\lib\site-packages\pandas\core\indexing.py:692: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy iloc._setitem_with_indexer(indexer, value, self.name) :18: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched2'] = val2 :19: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched3'] = val3
I supposed the passed rows into the enrich_row() function were Dataframes, so I tried to replace the "raw" assignments into it using the new assign() method of Dataframes:
passed_row.assign(enriched1 = val1)
passed_row.assign(enriched2 = val2)
passed_row.assign(enriched3 = val3)
But I got the following error:
AttributeError: 'Series' object has no attribute 'assign'
So the rows I'm passing to the function are Series.
Also, using Pandas dataframes directly with this code, these warnings do not appear.
I'm a little confused at this point. Any hint?