Row Sum of a each row in a Dataframe using Pyspark

user_1254579 0
Viewed 5238

There is a scenario of finding the sum of rows in a DF as follows

ID DEPT [..] SUB1 SUB2 SUB3 SUB4  **SUM1**
1  PHY      50    20   30   30   130
2  COY      52    62   63   34   211
3  DOY      53    52   53   84
4  ROY      56    52   53   74
5  SZY      57    62   73   54

Need to find row sum of SUB1 SUB2 SUB3 SUB4 for each rows and make as a new column SUM1. The ordinal position of the column SUB1 in the data frame is 16.

pyspark apache-spark python
1 Answers

You can use the Python sum to add up the columns:

import pyspark.sql.functions as F

col_list = ['SUB1', 'SUB2', 'SUB3', 'SUB4']
# or col_list = df.columns[16:20]

df2 = df.withColumn(
    'SUM1',
    sum([F.col(c) for c in col_list])
)
user_141657300
Related