How to increase the number of executors that process a certain stage in Spark Streaming/Kafka?

user_10282518 0
Viewed 755

I'm trying to read data from Kafka (1 topic, 5 partitions) using Spark Streaming Direct approach. In order to increase parallelism for processing the data, I set Dstream.repartition(60). However it seems that only 5 executors are running in each stage of each batch of streaming?

I had set these parameters:

--num-executors 20 
--executor-cores 4
--executor-memory 3G
--driver-memory 3G
--spark.default.Parallelism 60

My confusion is:
I want to decrease processing time, so I try to increase the number of Vcores, but this method doesn't seem to work. Why? Thanks.

spark-streaming apache-kafka apache-spark
1 Answers

(1 topic, 5 partitions) using Spark Streaming Direct approach.... It seems that only 5 executors are running

That's all you're going to get. You cannot have more than one consumer per topic partition.

The only way you'll get Spark (or any Kafka Consumer group) is to scale out more executors is to repartition the topic itself, and this is a different terminology than repartitioning in Spark

Note: increasing the number of partitions of a topic is possible, but the keys of the messages will no longer be one-to-one for the retention policy of the topic for the last message written before the repartition

After you repartition a topic, I believe you must restart the Spark application as it won't fetch new topic data dynamically

user_23086830
Related