Pyspark Py4JJavaError: An error occurred while calling o401.fit while trying to use IDF

user_7726586 0
Viewed 1247

I've setted up pyspark on google colab using this tutorial from towardsdatascience. It runs well until it fails on trying to use IDF

from pyspark.ml.feature import IDF

idf = IDF(inputCol='hash', outputCol='features')
model_idf = idf.fit(df_hash) <--- fails here

with next error:

Py4JJavaError: An error occurred while calling o401.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 56.0 failed 1 times, most recent failure: Lost task 0.0 in stage 56.0 (TID 697) (fbea1ac0124f executor driver): org.apache.spark.SparkException: Failed to execute user defined function(Tokenizer$$Lambda$3093/1092846241: (string) => array<string>)

Java version is 8

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"

Has anyone faced a similar problem?

pyspark google-colaboratory tf-idf python java
1 Answers

The problem was with NA in my data in the column which I tried to tokenize and then apply IDF. Dropping rows with NA in this column helped me:

df_without_na = df.na.drop(subset='my_column_name')
df_without_na.show()
user_77265860
Related