I have connected a kafka topic with 4 partitions into spark streaming using the new API: structured streaming. Everything goes well except the fact that out of my two worker nodes only one worker node (so one executor with 2 cores) is used. The jobs have 4 tasks but only one worker node is being used.
I tried to add number of executors in spark-submit but that didn't change anything.
Any ideas on how to engage the other nodes?
Thanks, Mikel