Say I have a null-containing DF as follows:
In [45]: df2.show()
+-----+----+
|name1| v|
+-----+----+
| john| 1.0|
| null| 4.0|
| meh|null|
+-----+----+
I'd like to add a label (for ML training), such that label is True if and only if name1 is john. This is what I'm doing:
In [47]: df3 = df2.withColumn('label', (df2.name1=='john'))
In [48]: df3.fillna(False, ['label']).show()
+-----+----+---------+
|name1| v| label|
+-----+----+---------+
| john| 1.0| true|
| null| 4.0| false|
| meh|null| false|
+-----+----+---------+
Is there a more idiomatic way of doing this? I found df2.withColumn('label', (df2.name1=='john') & ~df2.name1.isNull()) works, though I don't understand why it works.