I have a sample data which I would like to do the PySpark version of str.get_dummies but am not sure how to do it.
title_df:
Film Category
LOTR Drama, Fantasy
STAR WARS Sci-Fi, Action
etc.
Which will then give me with:
Film Action Fantasy Drama Sci-Fi Romance Comedy
LOTR 0 1 1 0 0 0
STAR WARS 1 0 0 1 0 0
etc.
Appreciate the help!
You can use crosstab:
import pyspark.sql.functions as F
df2 = (df.select('Film', F.explode(F.split('Category', ', ')).alias('Category'))
.distinct()
.crosstab('Film', 'Category')
)
df2.show()
+-------------+------+-----+-------+------+
|Film_Category|Action|Drama|Fantasy|Sci-Fi|
+-------------+------+-----+-------+------+
| LOTR| 0| 1| 1| 0|
| STAR WARS| 1| 0| 0| 1|
+-------------+------+-----+-------+------+
Or if you prefer classic Spark SQL operations:
df2 = (df.select('Film', F.explode(F.split('Category', ', ')).alias('Category'))
.groupBy('Film')
.pivot('Category')
.agg(F.lit(1))
.fillna(0)
)
This is the ML way, for example:
%python
from pyspark.ml.feature import VectorAssembler, StringIndexer, OneHotEncoder
stringIndexer = StringIndexer().setInputCol("ageCategory").setOutputCol("ageIndex")
ageModel = stringIndexer.fit(df)
df = ageModel.transform(df)