Pandas to Pyspark environment

Abhinav bharti 0
Viewed 29
newlist = []
for column in new_columns:
    count12 = new_df.loc[new_df[col].diff() == 1]
    new_df2=new_df2.groupby(['my_id','friend_id','family_id','colleage_id']).apply(len)

There is no option is available in pyspark for getting all length of column

How can we achieve this code into pyspark.

Thanks in advance..

dataframe pyspark apache-spark-sql python
1 Answers

Literally, apply(len) is just an aggregation function that would count grouped elements from groupby. You can do the very same thing in basic PySpark syntax

import pyspark.sql.functions as F

(df
    .groupBy('my_id','friend_id','family_id','colleage_id')
    .agg(F.count('*'))
    .show()
)
pltc0
Related