python pandas dataframe mutate all elements as (element / max element in row)

user_14790802 0
Viewed 228

df=

x1, x2, x3, x4, x5...x90, 'ignore'
37.4, 35.6, 43.5, 33.3, 42.4...42.5, 'ignoe'
32.2, 37.2, 43.4, 34.3, 82.4...22.1, 44
33.5, 45.5, 25.9, 23.8, 24.2...52.2, True
35.4, 30.6, 33.3, 33.3, 42.5...82.5, 'ignre'

I want to mutate all of the elements so they will all be between 0-1, by dividing each element by the max element in the row, ignoring the last column (there will not be any negative numbers or 0)

The problem I am having is it needs to be performant on millions of rows but iterating over rows in pandas is very slow

I believe this works but it is very slow

for index, row in df.iterrows():
    rowMax = max(row.drop(columns='ignore'))
    for i in range(1, 91):
        df.loc[index, 'x' + str(i)] = df.loc[index, 'x' + str(i)] / rowMax

output:

x1, x2, x3, x4, x5...x90, 'ignore'
0.86, 0.82, 1.00, 0.77, 0.97...0.98, 'ignoe'
pandas python
2 Answers
df.update(df.iloc[:,0:90].div(df.max(axis=1), axis=0))

First, you should grab the part of the dataframe you want to do the divide on: df.iloc[:,0:90] which means for every row pick the columns from 0 to 89.

Then, you grab the max of each row with: df.max(axis=1). Because your ignore column does not have all values as numerical, it would be ignored by max function so no worries about that.

Then, you do a divide with axis=0 meaning that divide on each row.

And, as the last step, you update the dataframe.

user_72067010

Try Using apply function link

Remove any str or object column and store it some another variable and concatenate it after the operation I created the the data frame of size

df.shape
Out[38]: (1000000, 91)

col = pd.DataFrame(df['col_str']) # Storing Categorical Col
df = df.drop('col_str', axis=1) # Dropping the col

start = datetime.now()    
tmp = df.apply(lambda x : x/max(x), axis=1, raw=True)
print(datetime.now()-start)
0:00:16.693436

It took around 16secs

user_63085980
Related