How to install kafka module in pyspark

user_12189846 0
Viewed 1151

I have problem when import KafkaUtils is : No module named 'pyspark.streaming.kafka' But i don't know how to install kafka module.

I use python 3.6.8, spark 2.2.0 and kafka_2.12-2.5.0

spark-streaming pyspark apache-kafka
1 Answers

As it turns out, KafkaUtils is being deprecated and replaced with Spark Structured Streaming. Which means you have two paths forward:

  1. Redesign your application to use Structured Streaming instead (see https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html for a primer).

  2. Downgrade your version of Spark to a version that still includes KafkaUtils as part of the distribution (you'll find that KafkaUtils won't need to be installed separately).

user_2164200
Related