Pyspark orderBy asc nulls last

user_2553191 0
Viewed 6064

In spark sql, you can use asc_nulls_last in an orderBy, eg

df.select('*').orderBy(column.asc_nulls_last).show

see Changing Nulls Ordering in Spark SQL.

How would you do this in pyspark?

I'm specifically using this to do a "window over" sort of thing:

df = df.withColumn(
    'rank',
    row_number().over(Window.partitionBy('group_id').orderBy('datetime'))
)

where the datetime column can be a datetime or null.

I was hoping to do it with:

...orderBy(expr('column asc NULLS last'))

But this errors with Exception: mismatched input 'NULLS' expecting <EOF>

pyspark apache-spark
1 Answers
from pyspark.sql import functions as F
df = df.withColumn(
    'rank',
    F.row_number().over(Window.partitionBy('group_id').orderBy(F.col('datetime').asc_nulls_last()))
)
user_33640690
Related