remove specific words into a dataframe with pyspark

Use split function to split the column by space then get the last element of array.

• from Spark2.4+ use element_at function
• for Spark < 2.4 use reverse(split(array))[0]

#using element_at
df.withColumn("formacion",element_at(split(col("formacion"),"\\s"),-1)).show() 

#or using array_index
df.withColumn("formacion",split(col("formacion"),"\\s")[1]).show()

#split reverse and get first index value
df.withColumn("formacion",reverse(split(col("formacion"),"\\s"))[0]).show()

#+-----+--------------+----------+-------------+
#|   id|titulo        |tipo      | formacion   |
#+------+--------------------+-----------------+
#|32084|A             | Material |  TRADE      |
#|32350|B             | Curso    |  LEADER     |
#|32362|C             | Curso    |  EVALUA     |   
#|32347|D             | Curso    |  CUMPLI     |      
#|32036|E             | Curso    |  INFORM     |  
#+-----+--------------+----------+-------------+

Sorry guys , this is original Column from DataFrame

formacion = [ VION00001 TRADE, CUS11222 LEADER,ITIN9876 EVALUA ,VION1234 CUMPLI,EVAN11 FR]

This is expected

formacion = [ TRADE, LEADER,EVALUA ,CUMPLI, FR]