How to assign a value to a column in Dask data frame

user_5117016 0
Viewed 1712

How to do the same as the bellow code for a dask data frame.

df['new_column'] = 0
for i in range(len(df)):
    if (condition):
        df[i,'new_column'] = '1'
    else:
        df[i,'new_column'] = '0'

I want to add a new column to a dask dataframe and insert 0/1 to the new column.

dask append insert python
3 Answers

In case you do not wish to compute as suggested by Rajnish kumar, you can also use something along the following lines:

import dask.dataframe as dd
import pandas as pd
import numpy as np

my_df = [{"a": 1, "b": 2}, {"a": 2, "b": 3}]
df = pd.DataFrame(my_df)
dask_df = dd.from_pandas(df, npartitions=2)
dask_df["c"] = dask_df.apply(lambda x: x["a"] < 2, 
                             axis=1, 
                             meta=pd.Series(name="c", dtype=np.bool))
dask_df.compute()

Output:

    a   b   c
0   1   2   True
1   2   3   False

The condition (here a check whether the entry in column "a" < 2) is applied on a row-by-row-basis. Note that depending on your condition and dependencies therein it might not necessarily be as straightforward, but in that case you could share additional information on what your condition entails.

user_20749880

You can't do that directly to Dask Dataframe. You first need to compute it. Use this, It will work.

df = df.compute()
for i in range(len(df)):
if (condition):
    df[i,'new_column'] = '1'
else:
    df[i,'new_column'] = '0'

The reason behind this is Dask Dataframe is the representation of dataframe schema, it is divided into dask-delayed task. Hope it helps you.

user_75925150

I was going through these answers for a similar problem I was facing.

This worked for me.

def extractAndFill(df, datetimeColumnName):
  # Add 4 new columns for weekday, hour, month and year
  df['pickup_date_weekday'] = 0 
  df['pickup_date_hour'] = 0
  df['pickup_date_month'] = 0
  df['pickup_date_year'] = 0

  # Iterate through each row and update the values for weekday, hour, month and year
  for index, row in df.iterrows():
    # Get weekday, hour, month and year
    w, h, m, y = extractDateParts(row[datetimeColumnName])

    # Update the values
    row['pickup_date_weekday'] = w
    row['pickup_date_hour'] = h
    row['pickup_date_month'] = m
    row['pickup_date_year'] = y

  return df

df1.compute()
df1 = extractAndFill(df1, 'pickup_datetime')
user_37199320
Related