Provided Maven Coordinates must be in the form 'groupId:artifactId:version' PySpark and Kafka

user_4473615 0
Viewed 1902

While converting Kafka messages to dataframe am getting error while passing the packages as an argument. 

from pyspark.sql import SparkSession, Row
from pyspark.context import SparkContext
from kafka import KafkaConsumer
import os

os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0.jar: org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:jar:2.1.1 pyspark-shell'pyspark-shell'

sc = SparkContext.getOrCreate()
spark = SparkSession(sc)

df = spark \
  .read \
  .format("kafka") \
  .option("kafka.bootstrap.servers", "localhost:9092") \
  .option("subscribe", "Jim_Topic") \
  .load()
df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")

Error ::::::::::::::::::::::::::::::::::::::::::::::

    ::          UNRESOLVED DEPENDENCIES         ::

    ::::::::::::::::::::::::::::::::::::::::::::::

    :: org.apache.spark#spark-sql-kafka-0-10_2.11;2.2.0.jar: not found

    ::::::::::::::::::::::::::::::::::::::::::::::
pyspark apache-kafka apache-spark
1 Answers

As the exception suggests, you have a typo in one of your dependencies.

org.apache.spark:spark-sql-kafka-0-10_2.12:jar:

is missing the version (and it also has an unecessary :). The following should do the trick: 

org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0

The full dependencies will become:

'--packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.1.1'
user_71317570
Related