I am quite new to pyspark. I am trying to filter a pyspark dataframe on dates iteratively. I am trying to create some function that can filter through dates to query one week's data for each month in a given date range.
My input table looks something like this and this is partitioned on date
| date | user_id |
|---|---|
| 2018-01-01 | abc |
| 2018-01-02 | abc |
| 2018-01-02 | xyz |
| 2018-01-02 | ghk |
So I would ideally define: start_date == '2018-01-01' & end_date == '2021-12-31' and create a filtered dataframe.
I am having trouble creating a loop that I can use as a filter. Can someone help please!