Fetch week start date and week end date from Date

user_5370631 0
Viewed 7069

I need to fetch week start date and week end date from a given date, taking into account that the week starts from Sunday and ends on Saturday.

I referred this post but this takes monday as starting day of week. Is there any inbuilt function in spark which can take care of this?

pyspark apache-spark-sql
3 Answers

Find out the day of the week and Use selectExpr to iterate through columns , and making Sunday as week start date

from pyspark.sql import functions as F


df_b = spark.createDataFrame([('1','2020-07-13')],[ "ID","date"])
df_b = df_b.withColumn('day_of_week', F.dayofweek(F.col('date')))
df_b = df_b.selectExpr('*', 'date_sub(date, day_of_week-1) as week_start')
df_b = df_b.selectExpr('*', 'date_add(date, 7-day_of_week) as week_end')

df_b.show()

+---+----------+-----------+----------+----------+
| ID|      date|day_of_week|week_start|  week_end|
+---+----------+-----------+----------+----------+
|  1|2020-07-13|          2|2020-07-12|2020-07-18|
+---+----------+-----------+----------+----------+

Update in Spark SQL

Crete a Temporary view from the data-frame first

df_a.createOrReplaceTempView("df_a_sql")

Code here

%sql
select *, date_sub(date,dayofweek-1) as week_start,
date_sub(date, 7-dayofweek) as week_end
from
(select *, dayofweek(date) as dayofweek
from df_a_sql) T

Output

+---+----------+-----------+----------+----------+
| ID|      date|day_of_week|week_start|  week_end|
+---+----------+-----------+----------+----------+
|  1|2020-07-13|          2|2020-07-12|2020-07-18|
+---+----------+-----------+----------+----------+
user_51082200

Perhaps this is helpful -

Load the test data

   val df = spark.sql("select cast('2020-07-12' as date) as date")
    df.show(false)
    df.printSchema()

    /**
      * +----------+
      * |date      |
      * +----------+
      * |2020-07-15|
      * +----------+
      *
      * root
      * |-- date: date (nullable = true)
      */

week starting from SUNDAY and ending SATURDAY


    // week starting from SUNDAY and ending SATURDAY
    df.withColumn("week_end", next_day($"date", "SAT"))
      .withColumn("week_start", date_sub($"week_end", 6))
      .show(false)

    /**
      * +----------+----------+----------+
      * |date      |week_end  |week_start|
      * +----------+----------+----------+
      * |2020-07-12|2020-07-18|2020-07-12|
      * +----------+----------+----------+
      */

week starting from MONDAY and ending SUNDAY


    // week starting from MONDAY and ending SUNDAY
    df.withColumn("week_end", next_day($"date", "SUN"))
      .withColumn("week_start", date_sub($"week_end", 6))
      .show(false)

    /**
      * +----------+----------+----------+
      * |date      |week_end  |week_start|
      * +----------+----------+----------+
      * |2020-07-12|2020-07-19|2020-07-13|
      * +----------+----------+----------+
      */

week starting from TUESDAY and ending MONDAY

    // week starting from TUESDAY and ending MONDAY
    df.withColumn("week_end", next_day($"date", "MON"))
      .withColumn("week_start", date_sub($"week_end", 6))
      .show(false)

    /**
      * +----------+----------+----------+
      * |date      |week_end  |week_start|
      * +----------+----------+----------+
      * |2020-07-12|2020-07-13|2020-07-07|
      * +----------+----------+----------+
      */
user_47588230

Find out the start date and end date of week in pyspark dataframe. Monday being the first day of week.

def add_start_end_week(dataframe, timestamp_col, StartDate, EndDate):
""""
Function:
    Get the start date and the end date of week
args
    dataframe: spark dataframe
    column_name: timestamp column based on which we have to calculate the start date and end date
    StartDate: start date column name of week
    EndDate: end date column name of week
"""
dataframe = dataframe.withColumn(
    'day_of_week', dayofweek(col(timestamp_col)))
# start of the week (Monday as first day)
dataframe = dataframe.withColumn('StartDate',when(col("day_of_week")>1, \
                                                  expr("date_add(date_sub({},day_of_week-1),1)".format(timestamp_col))). \
                                                  otherwise(expr("date_sub({},6)".format(timestamp_col))))
#End of the Week
dataframe = dataframe.withColumn('EndDate',when(col("day_of_week")>1, \
                                                expr("date_add(date_add({},7-day_of_week),1)".format(timestamp_col))). \
                                                otherwise(col("{}".format(timestamp_col))))

return dataframe

Validate the above function:

df = spark.createDataFrame([('2021-09-26',),('2021-09-25',),('2021-09-24',),('2021-09-23',),('2021-09-22',),('2021-09-21',),('2021-09-20',)], ['dt'])
dataframe = df.withColumn('day_of_week', dayofweek(col('dt')))
# start of the week (Monday as first day)
dataframe = dataframe.withColumn('StartDate',when(col("day_of_week")>1,expr("date_add(date_sub(dt,day_of_week-1),1)")).otherwise(expr("date_sub(dt,6)")))
#End of the Week
dataframe = dataframe.withColumn('EndDate',when(col("day_of_week")>1,expr("date_add(date_add(dt,7-day_of_week),1)")).otherwise(col("dt")))
user_71073960
Related