I have a Dataframe called df with around 20m rows, that looks like
userId movieId rating
0 1 296 5.0
1 1 306 3.5
2 1 307 5.0
3 2 665 5.0
4 2 899 3.5
...
and I have a Series, user_bias
userId
1 0.280431
2 0.096580
3 0.163554
4 -0.155755
5 0.218621
...
I would like to subtract the matching value according to userId column in user_bias from df['rating']. For example the rating value of the first row should be replaced with 5.0 - 0.280431 = 4.719569. I tried two solutions but they seems to be very slow. Is there a better way to achieve this?
Solution 1
for i, row in df.iterrows():
df.at[i, 'rating'] -= user_bias[row.userId]
Solution 2
To get rid of the for loop, I've used apply method. Not sure if it is correct result-wise but it is again way slower than I expected.
df['rating'] = df.apply(lambda row: row.rating - user_bias[row.userId], axis=1)