"A value is trying to be set on a copy of a slice from a DataFrame" for Series objects using Dask?

Viewed 211

I'm testing the apply() method of Dask dataframes and I'm running this code:

import pandas as pd
import dask.dataframe as dd
import time


def enrich_str(str):
        
    val1 = f'{str}_1'
    val2 = f'{str}_2'
    val3 = f'{str}_3'
    time.sleep(3)
    
    return val1, val2, val3
    
def enrich_row(passed_row):
    
    col_name = str(passed_row['colName'])
    my_string = str(passed_row[col_name])
    
    val1, val2, val3 = enrich_str(my_string)
    
    passed_row['enriched1'] = val1
    passed_row['enriched2'] = val2
    passed_row['enriched3'] = val3
    
    return passed_row

df = pd.DataFrame({'numbers': [1, 2, 3, 4, 5], 'colors': ['red', 'white', 'blue', 'orange', 'red']}, 
                  columns=['numbers', 'colors'])
ddf = dd.from_pandas(df, npartitions=2)

ddf['colName'] = 'colors'

result = ddf.apply(enrich_row, axis=1,
                   meta={'numbers': 'int64', 'colors': 'string', 'colName': 'string',
                         'enriched1': 'string', 'enriched2': 'string', 'enriched3': 'string'})

tic = time.perf_counter()
enriched_df = result.compute()
toc = time.perf_counter()

print(f"{enriched_df.shape[0]} rows enriched in {toc - tic:0.4f} seconds")

print(enriched_df)

The final result is correct, but I'm getting these warnings:

5 rows enriched in 9.0715 seconds :17: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched1'] = val1 C:\Users\LZavarella\miniconda3\envs\pbi_powerquery_env\lib\site-packages\pandas\core\indexing.py:692: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy iloc._setitem_with_indexer(indexer, value, self.name) :18: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched2'] = val2 :19: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy passed_row['enriched3'] = val3

I supposed the passed rows into the enrich_row() function were Dataframes, so I tried to replace the "raw" assignments into it using the new assign() method of Dataframes:

passed_row.assign(enriched1 = val1)
passed_row.assign(enriched2 = val2)
passed_row.assign(enriched3 = val3)

But I got the following error:

AttributeError: 'Series' object has no attribute 'assign'

So the rows I'm passing to the function are Series.

Also, using Pandas dataframes directly with this code, these warnings do not appear.

I'm a little confused at this point. Any hint?

1 Answers

I'm running your code on MacOS, but unable to reproduce the problem with the following versions:

- python=3.9.1
- pandas=1.2.4
- dask=2021.4.1
Related