How to dynamically allocate resources and manage spike for spark structured streaming?

user_4111019 0
Viewed 837

I have written a structured streaming app to push data real time to S3. We receive data from Kafka and push data to S3 after processing data using structured streaming.

I know Spark streaming supports dynamic allocation of resources. But found following JIRA https://issues.apache.org/jira/browse/SPARK-24815 that says dynamic allocation of resource is not supported by structured streaming.

Want to know to handle spike in data when dynamic allocation of resources is not currently supported for structured streaming.

Reasoning for NOT implementing Dynamic Allocation is Following (From JIRA): If we set spark.dynamicAllocation.enabled=true and run a structured streaming job, the batch dynamic allocation algorithm kicks in. It requests more executors if the task backlog is a certain size, and removes executors if they idle for a certain period of time. It is basically meant for batch processing not continuous processing. In batch mode we care about the numRunningOrPendingTasks while for streaming about the ratio: averageBatchProcTime.toDouble / batchDurationMs i.e. average batch processing time per unit of time, so scaling the system down when idle is not a valid solution.

Question: How to handle SPIKES when dynamic allocation of resources is NOT provided.

spark-structured-streaming apache-spark
1 Answers

So, by setting minPartitions to a number higher than topics partitions, lets say we have 10 partitions in the topic, and we set minPartitions=1000, number of tasks scales up and down to meet the Trigger interval. However, when the load comes down, Kafka source still considers consumer cache and round-robins the tasks on all executors. So executors never remain idle to scale down by dynamicAllocation idle time.

user_181165570
Related