spark on yarn,only one executor on one node works and the allocation is random

Viewed 614

i run spark-shell with this command:

./bin/spark-shell --master yarn --num-executors 16       
--executor-memory 14G --executor-cores 8

i have four nodes,every node has 16G memory and 4cores

after i changed num-executors,the spark webUi tell me it worked, enter image description here BUT only one executor on node which named "slave" is running

we could see TRANSFER1 and TRANSFER2 is empty enter image description here

how can i solve this, when i submit job the situation has not changed should i change worker-instances or num-executors and so on

0 Answers
Related