How to iterate dataframe´s index and columns in comprehension list?

user_7669782 0
Viewed 900

If I need to iterate a df in a comprehension list, I would do this:

df['new_col'] = [x if y == 1 and z == 2 for x,y in df[['col_1', 'col_2']].values]

If instead of iterating col_1 and col_2, I need to iterate df.index and df_col_2 values?

What is the syntax inside the comprehension list in this for conditional example?

pandas list-comprehension python
2 Answers

pandas dataframe and series have iteration methods. So to iterate over index and a given column you can use iteritems:

df['new_col'] = [x if y == '1' and z =='2' for x, y in df['col_2'].iteritems()]

In this case x is the index and y the value of column col2

More generally iterrows gives you access to index and all columns in one iteration:

for idx, row in df.iterrows():
    print("Index", idx)
    print("col1", row.col1)
    print("col2", row.col2)
    ...
user_74375240

Let's take an example of the dataframe. Then look at various things you can do with df.loc and df.index.

I will take a simple example of 5 kids ages 1 thru 4, and the points they have earned so far. Normally, I would prefer unique indexes but for this example we want you want to search by index. So I am making it nonunique.

import pandas as pd
df = pd.DataFrame({'Points': [2, 4, 8, 3, 2, 5, 6],
                   'Age': [2, 3, 4, 1, 2, 4, 3]},
                  index=['Bob', 'Mike', 'Steve', 'Kate', 'Jane', 'Jill', 'Jane'])
                  
print (df)

The DataFrame will look like this:

       Points  Age
Bob         2    2
Mike        4    3
Steve       8    4
Kate        3    1
Jane        2    2
Jill        5    4
Jane        6    3

If we want to find all the kids with Age = 2 and name = Jane, you can give:

x = df.loc[(df.Age == 2) & (df.index == 'Jane'), 'Points'].tolist() #or .values

print (x)

Output will be:

[2]

If you want to find all the kids with Age = 2 and have scored 2 points, you can give:

y = df.index[(df.Age == 2) & (df.Points == 2)].tolist() #or .values

print (y)

Output will be:

['Bob', 'Jane']

If you want to find all the kids with Age = 3, you can give:

z = df.index[(df.Age == 3)].tolist() #or .values

print (z)

Output will be:

['Mike', 'Jane']

Let's say you want to narrow this down and update Jane aged 2 with Found in New_col, then you can give:

df.loc[(df.Age == 2) & (df.index == 'Jane'), 'New_col'] = 'Found'

print (df)

The outputs will be:

       Points  Age New_col
Bob         2    2     NaN
Mike        4    3     NaN
Steve       8    4     NaN
Kate        3    1     NaN
Jane        2    2   Found
Jill        5    4     NaN
Jane        6    3     NaN

None of these required us to iterate through the dataframe using df.iterrows(). We can do a lot of these data manipulation without the use of df.iterrows(). If you have a specific usecase, let's review it and come up with a solution that does not involve iteration.

user_138739800
Related