Why is there a syntax error on pyspark broadcast call

wram19 0
Viewed 24

I'm following a textbook example about creating a music recommender and when trying to set up the alias connection to link incorrect names to the correct names in the artist name column, I'm getting a syntax error.

The error is on line 3 and its on the broadcast call. It has an issue with the how='left'

from pyspark.sql.functions import broadcast, when

train_data = user_artist_df.join(broadcast(artist_alias),'artist', how='left').\ //here is the error

train_data = train_data.withColumn('artist',
                                        when(col('alias').isNull(), col('artist')).\
                                        otherwise(col('alias')))

train_data = train_data.withColumn('artist', col('artist').\
                                                cast(IntegerType())).\
                                                drop('alias')

train_data.cache()

train_data.count()
pyspark
1 Answers

You have a ./ at the end of line 3 that is causing the error. This should work:

from pyspark.sql.functions import broadcast, when

train_data = user_artist_df.join(broadcast(artist_alias),'artist', how='left')

train_data = train_data.withColumn('artist',
                                        when(col('alias').isNull(), col('artist')).\
                                        otherwise(col('alias')))

train_data = train_data.withColumn('artist', col('artist').\
                                                cast(IntegerType())).\
                                                drop('alias')

train_data.cache()

train_data.count()
Robert Kossendey0
Related