Disclaimer: I've very new to pyspark
I've grouped my dataframe on the id and month_year to get total counts of how much a user engages with your product. Now this issue is months with no activity are now showing up my final df.
This is how its done:
app_sessions_per_month = app_sessions.where("session_start_date_pt > '2019-05-25'").groupby('id','month_year').agg(F.sum('action').alias('count'))
+--------------------+----------+----------------------+
| core_id|month_year|month_sum_detailaction|
+--------------------+----------+----------------------+
|aa01bb6f-2dd8-43e...| 7_2019| 0|
|aa01bb6f-2dd8-43e...| 9_2019| 0|
|aa01bb6f-2dd8-43e...| 10_2019| 0|
+--------------------+----------+----------------------+
The dates actually go from 05_2019 to 05_2020.
Original DF:
id month_year count
1. 02_2020. 1
1. 03_2020. 4
1. 05_2020. 2
1. 06_2020. 7
1. 07_2020. 2
Desired DF:
id month_year count
1. 01_2020. 0
1. 02_2020. 1
1. 03_2020. 4
1 04_2020. 0
1. 05_2020. 2
1. 06_2020. 7
1. 07_2020. 2
Now this example just shows 1 id but my actual dataset has millions of ids
Any help would be much appreciated.