Add a range array column in pypsark

Viewed 41

Range Array Column

I want to add an array column with the range/sequence of number of days between the start date and end date. An example of the data would be

from datetime import datetime
from pyspark.sql.types import *
from pyspark.sql import functions as F

data = [
    (
        datetime(2020,1,1),
        datetime(2020,12,31),
    ),
    
]
rdd = sc.parallelize(data)

schema = StructType(
    [
       StructField("dateStart", DateType(), True),
       StructField("dateEnd", DateType(), True)
    ]
)

df = spark.createDataFrame(rdd, schema)

df = df.withColumn(
    "dateDiff",
    F.datediff(
        F.col("dateEnd"),
        F.col("dateStart"),
    )
)

df.show()
+----------+----------+--------+
| dateStart|   dateEnd|dateDiff|
+----------+----------+--------+
|2020-01-01|2020-12-31|     365|
+----------+----------+--------+

and what I need is the following so that I can explode the range column.

+----------+----------+--------+------------+
| dateStart|   dateEnd|dateDiff|       range|
+----------+----------+--------+------------+
|2020-01-01|2020-12-31|     365| [1,2,..365]|
+----------+----------+--------+------------+

I wrote a udf to do this, but I want to know if there is a more efficient way to do this?

def range_func(x):
    return list(range(1, x+1))

range_udf = F.udf(range_func, ArrayType(IntegerType()))

df = df.withColumn("Range", range_udf(F.col("dateDiff")))
df.show()
+----------+----------+--------+--------------------+
| dateStart|   dateEnd|dateDiff|               Range|
+----------+----------+--------+--------------------+
|2020-01-01|2020-12-31|     365|[1, 2, 3, 4, 5, 6...|
+----------+----------+--------+--------------------+
1 Answers

for spark version 2.4+, use sequence function

df = df.withColumn('Range', F.sequence(F.lit(1), F.col('dateDiff')+1))
df.show()

# +----------+----------+--------+--------------------+
# | dateStart|   dateEnd|dateDiff|               Range|
# +----------+----------+--------+--------------------+
# |2020-01-01|2020-12-31|     365|[1, 2, 3, 4, 5, 6...|
# +----------+----------+--------+--------------------+
Related