Split PySpark dataframe column at the dot

user_9629771 0
Viewed 3250

I have tried the below in Pandas and it works. I wondered how I might do it in PySpark?

The input is 

news.bbc.co.uk

it should split it at the '.' and hence index should equal:

[['news', 'bbc', 'co', 'uk'], ['next', 'domain', 'name']]

index = df2.domain.str.split('.').tolist()

Does anyone know how I'd do this in spark rather than pandas?

Thanks

pyspark apache-spark python-3.x
3 Answers

Using '.' works in a different way. Using it with escape character '\' actually worked.

df = df.withColumn('col_name', F.split(F.col('col_name'), '\.'))
user_116304780

You can use pyspark.sql.functions.split to split str.

import pyspark.sql.functions as F

df = df.withColumn('col_name', F.split(F.col('col_name'), '.'))
user_70459870 
df.select(split("col_name", '[\.]'))

or

df.selectExpr("split(col_name, '[\.]')")
user_50634390
Related