When using df_hdr_join.count() > 0 in when statement, it gives an error 'condition should be a Column'. I tried following.
df_result = df.withColumn('NUM', when(df_lbr_join.count() > 0, lit(monotonically_increasing_id())).otherwise(when(df_hdr_join.count() > 0, lit(monotonically_increasing_id())).otherwise(lit(None))))
I need to check the size of the dataframe inside when statement and increment the value. How this should be done?