Error: Spark streaming- Kafka :py4j.protocol.Py4JJavaError: An error occurred while calling o80.load

Viewed 22

I am trying out with this code but an error occurs and I have used many jar files but couldn't resolve may I know how to solve this.

from pyspark.sql import SQLContext 
from pyspark import SparkContext 
from pyspark.sql import SparkSession 

spark = SparkSession.builder.config("").appName("streaming").getOrCreate() 
df = spark.readStream.format("kafka") \
  .option("kafka.bootstrap.servers", "localhost:001") \
  .option("subscribe", "sourcefoobar") \
  .option("includeHeaders", "true") \
  .option("driver", '/home/akash/mysql-connector-java-8.0.29.jar') \ 
  .load() 
df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")

I have encountered this error:

py4j.protocol.Py4JJavaError: An error occurred while calling o80.load. : java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider

0 Answers
Related