How to check if there is a row with same value combinations in a dataframe?

Viewed 130

I have a dataframe and want to create a new column based on other rows of the dataframe. My dataframe looks like

    MitarbeiterID   ProjektID   Jahr    Monat   Week    mean    freq    last
0       583          83224      2020    1        2      3.875    4       0
1       373          17364      2020    1        3      5.00     0       4
2       923          19234      2020    1        4      5.00     3       3
3       643          17364      2020    1        3      4.00     2       2

Now I want to check, if the freq of a row is zero, then I will check if there is another row with the same ProjektID and Year an Week where the freq is not 0. If this is true I want a new column "other" which is value 1 and 0 else. So, the output should be

    MitarbeiterID   ProjektID   Jahr    Monat   Week    mean    freq    last other
0       583          83224      2020    1        2      3.875    4       0     0
1       373          17364      2020    1        3      5.00     0       4     1
2       923          19234      2020    1        4      5.00     3       3     0
3       643          17364      2020    1        3      4.00     2       2     0

This time I have no approach, can anyone help? Thanks!

3 Answers

I think the best way to solve this is not to use pandas too much :-) converting things to sets and tuples should make it fast enough.

The idea is to make a dictionary of all the triples (ProjektID, Jahr, Week) that appear in the dataset with freq != 0 and then check for all lines with freq == 0 if their triple belongs to this dictionary or not. In code, I'm creating a dummy dataset with:

x = pd.DataFrame(np.random.randint(0, 2, (8, 4)), columns=['id', 'year', 'week', 'freq'])

which in my case randomly gave:

>>> x
   id  year  week  freq
0   1     0     0     0
1   0     0     0     1
2   0     1     0     1
3   0     0     1     0
4   0     1     0     0
5   1     0     0     1
6   0     0     1     1
7   0     1     1     0

Now, we want triplets only where freq != 0, so we use

x1 = x.loc[x['freq'] != 0]
triplets = {tuple(row) for row in x1[['id', 'year', 'week']].values}

Note that I'm using x1.values, which is not a pandas DataFrame but rather a numpy array; so each row in there can now be converted to tuple. This is necessary because dataframe rows, or even numpy array or lists, are mutable objects and cannot be hashed in a dictionary otherwise. Using a set instead of e.g. a list (which doesn't have this restriction) is for efficiency purposes.

Next, we define a boolean variable which is True if a triplet (id, year, week) belongs to the above set:

belongs = x[['id', 'year', 'week']].apply(lambda x: tuple(x) in triplets, axis=1)

We are basically done, this is the further column you want, except for also needing to force freq == 0:

x['other'] = np.logical_and(belongs, x['freq'] == 0).astype(int)

(the final .astype(int) is to have it values 0 and 1, as you were asking, instead of False and True). Final result in my case:

>>> x
   id  year  week  freq  other
0   1     0     0     0      1
1   0     0     0     1      0
2   0     1     0     1      0
3   0     0     1     0      1
4   0     1     0     0      1
5   1     0     0     1      0
6   0     0     1     1      0
7   0     1     1     0      0

The following solution tests if the required conditions are True.

import io
import pandas as pd

Data

df = pd.read_csv(io.StringIO("""
MitarbeiterID   ProjektID   Jahr    Monat   Week    mean    freq    last
0       583          83224      2020    1        2      3.875    4       0
1       373          17364      2020    1        3      5.00     0       4
2       923          19234      2020    1        4      5.00     3       3
3       643          17364      2020    1        3      4.00     2       2
"""), sep="\s\s+", engine="python")

Make a column other with all values zero.

df['other'] = 0

If ProjektID, Jahr, Week are duplicated and any of the Freq values is larger than zero, then the rows that are duplicated (keep=False to also capture the original duplicated row) and where Freq is zero will have the value Other filled with 1. Change any() to all() if you need all values to be larger than zero.

if (df.loc[df[['ProjektID','Jahr', 'Week']].duplicated(), 'freq'] > 0).any(): df.loc[(df[['ProjektID','Jahr', 'Week']].duplicated(keep=False)) & (df['freq'] == 0), ['other']] = 1
else: print("Other stays zero")

Output:

enter image description here

Looks like I am too late ...:

df.set_index(['ProjektID', 'Jahr', 'Week'], drop=True, inplace=True)
df['other'] = 0
df.other.mask(df.freq == 0,
              df.freq[df.freq == 0].index.isin(df.freq[df.freq != 0].index),
              inplace=True)
df.other = df.other.astype('int')
df.reset_index(drop=False, inplace=True)
Related