I'm trying to split strings in a pyspark dataframe column with names and titles separated by different delimiters, in different formats. Eg:
+---+------------------------------------------------------------------------------------+
| id| Text |
| 0| first name last name (title), first name last name (title), and first name (title) | |
| 1| title: first name last name title: first name last name |
| 2| first name last name: title. first name last name: title. |
How can I get this into a column of just the names?
Desired output
| id| Text |
| 0| first name last name | |
| 1| first name last name |
| 2| first name last name |
I tried the following but it didn't work
test = table.withColumn('column_new', f.split(f.regexp_replace('column', '[(:;)]', r"$1,"), ","))
Thanks in advance.