Add/Substract datetime in pyspark.pandas

user_10502606 0
Viewed 244

I got an error in calculating the date using pyspark.pandas. Is there any way to calculate the date with pyspark.pandas?

import pyspark.pandas as ps
import pandas as pd

df = pd.DataFrame({'year': [2015, 2016],
                   'month': [2, 3],
                   'day': [4, 5]})
df = ps.DataFrame(df)
srs = ps.to_datetime(df)
srs + timedelta(days=3)
# this yields the same error
srs.add(timedelta(days=3))
# this yields the same error
srs + pd.TimeDelta(days=3)

this yields this error TypeError: Addition can not be applied to datetimes.

while below works

srs = srs.to_pandas()
srs + timedelta(days=3)
pyspark pandas databricks datetime python
2 Answers

I did have a similar problem on pyspark==3.2.1, and this seemed to be the only solution, as like

(
    ps.to_datetime(pd.Series(['2015-02-04', '2016-03-05']))
    .apply(lambda single_dt: single_dt + pd.Timedelta(days=3))
)

Newer versions of Pyspark have to_timedelta function which solves this problem nicely too. As per here

user_25963630

according to spark docs :

pandas users can access to full pandas API by calling DataFrame.to_pandas(). pandas-on-Spark DataFrame and pandas DataFrame are similar. However, the former is distributed and the latter is in a single machine. When converting to each other, the data is transferred between multiple machines and the single client machine.

so convert spark data frame to pandas dataframe first:

import pyspark.pandas
from datetime import timedelta

df = pd.DataFrame({'year': [2015, 2016],
                   'month': [2, 3],
                   'day': [4, 5]})
df = df.to_pandas() # here 
df = df.to_datetime(df)
df + timedelta(days=3)
# this yield same error
df.add(timedelta(days=3))
user_13674540
Related