I am trying out with this code but an error occurs and I have used many jar files but couldn't resolve may I know how to solve this.
from pyspark.sql import SQLContext
from pyspark import SparkContext
from pyspark.sql import SparkSession
spark = SparkSession.builder.config("").appName("streaming").getOrCreate()
df = spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers", "localhost:001") \
.option("subscribe", "sourcefoobar") \
.option("includeHeaders", "true") \
.option("driver", '/home/akash/mysql-connector-java-8.0.29.jar') \
.load()
df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)", "headers")
I have encountered this error:
py4j.protocol.Py4JJavaError: An error occurred while calling o80.load. : java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider