PySpark throws ImportError, but Module actually exists and works well

user_8290444 0
Viewed 4154

I am using Cloudera and the Spark version is 2.1.0.

I was trying to crossJoin two tables and create a column with fuzzy match ratio (thus I need to import fuzzywuzzy). Here is the code:

from fuzzywuzzy import fuzz
def fuzzy_ratio(x,y):
    from fuzzywuzzy import fuzz
    res = fuzz.token_set_ratio(x,y)
    return res

fuzz_udf = F.udf(fuzzy_ratio,IntegerType())  # register UDF

Master = tableA.crossJoin(tableB) \
               .withColumn('ratio',fuzz_udf(tableA['colA'],tableB['colB']))

And it throws

ImportError: No module named fuzzywuzzy

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:144)
at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:87)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796)

But fuzzy.token_set_ratio works when I input it in the interactive shell. So I really don't know what's going on here.

Could anybody please help with my question? Thanks a million!

cloudera pyspark-sql fuzzywuzzy pyspark user-defined-functions
2 Answers

I was getting similar error and I added the fuzzywuzzy dependency to my spark submit command.

First zip the two package folders.In my case, the folders are at location C:\Anaconda2\Lib\site-packages\fuzzywuzzy and C:\Anaconda2\Lib\site-packages\fuzzywuzzy-0.15.0.dist-info

I then added the zip file to spark-submit command as ../bin/spark-submit /path/to/mycode.py --py-files /path/to/fuzzywuzzy.zip

This makes sure that all worker nodes get the fuzzywuzzy dependency.

There is also an option to add this dependency later in sparkContext as spark.sparkContext.addPyFile('/path/to/fuzzywuzzy.zip')

user_18854270
Related