How to divide dataframe row's each value by row's total sum (data normalization) in pyspark?

user_13161972 0
Viewed 489

I have a dataframe of user's preferences:

+-------+-----+-----+-----+
|user_id|Movie|Music|Books|
+-------+-----+-----+-----+
|   100 |  0  |  1  |  2  |
|   101 |  3  |  1  |  4  |
+-------+---------+-------+

How to 1) compute total sum of each row(user); 2) divide each value by that sum? so I get normalized preference values:

+-------+---- -+-------+-------+
|user_id| Movie| Music | Books |
+-------+----- +-------+-------+
|   100 |  0   | 0.33..| 0.66..|
|   101 |0.42..| 0.15..| 0.57..|
+-------+------+-------+-------+
pyspark
1 Answers
# get column names that need to be normalized
cols = [col for col in df.columns if col != 'user_id']

# sum the columns by row
rowsum = sum([df[x] for x in cols])

# select user_id and normalize other columns by rowsum
df.select('user_id', *((df[x] / rowsum).alias(x) for x in cols)).show()

+-------+-----+------------------+------------------+
|user_id|Movie|             Music|             Books|
+-------+-----+------------------+------------------+
|    100|  0.0|0.3333333333333333|0.6666666666666666|
|    101|0.375|             0.125|               0.5|
+-------+-----+------------------+------------------+
user_49834500
Related