Adding column to PySpark DataFrame depending on whether column value is in another column

Viewed 8810

I have a PySpark DataFrame with structure given by

[('u1', 1, [1 ,2, 3]), ('u1', 4, [1, 2, 3])].toDF('user', 'item', 'fav_items')

I need to add a further column with 1 or 0 depending on whether 'item' is in 'fav_items' or not.

So I would want

[('u1', 1, [1 ,2, 3], 1), ('u1', 4, [1, 2, 3], 0)]

How would I look up for second column into third column to decide value and how would I then add it?

2 Answers
Related