Pyspark to_date() function gives different answers on Windows and WSL Ubuntu

user_1790500 0
Viewed 158

I have a function that converts an int to a date which is then fed into datediff to find how many days since an event happened. One of our tests passes on PySpark Windows and in our Azure DevOps pipeline, but fails when run on PySpark in WSL Ubuntu. We've narrowed it down to the to_date() function producing different results on the 2 platforms, but don't understand why.

import pyspark.sql.functions as F
import datetime
def from_int_to_date(int_date: int) -> datetime.datetime:
    """
    Convert an integer in YYYYMMDD format into a datetime object
    """
    return datetime.datetime.strptime(str(int_date), "%Y%m%d")

If I calculate F.to_date(F.lit(from_int_to_date(20190401))) I get Column<b"to_date(TIMESTAMP '2019-04-01 00:00:00')"> on Windows and Column<b"to_date(TIMESTAMP('2019-03-31 23:00:00.0'))> on the version running under WSL.

I am based in the UK and on 1 April 2019 we did our clock change for summer so I can understand the reason why it goes back an hour as the problem doesn't occur with an input int of 20190331. I'm just trying to understand why the behaviour of to_date() is different on the two systems and what we should do to mitigate for this (and any other differences) as ideally our code would be platform agnostic.

pyspark windows-subsystem-for-linux datetime windows
1 Answers

Set the timezone to the spark driver with the configuration spark.sql.session.timeZone so you won't depend on the system clock.

spark.conf.set("spark.sql.session.timeZone", "Europe/London")

This option can be settled even when the spark session is created.

user_34372020
Related