EMR Spark - TransportClient: Failed to send RPC

user_4319301 0
Viewed 22853

I'm getting this error, I tried to increase memory on cluster instances and in the executor and driver parameters without success.

17/05/07 23:17:07 ERROR TransportClient: Failed to send RPC 6465703946954088562 to ip-172-30-12-164.eu-central-1.compute.internal/172.30.12.164:34706: java.nio.channels.ClosedChannelException

Does anyone have any clue to fix this error?

BTW I'm using YARN as cluster manager

Thanks in advance

emr hadoop-yarn apache-spark
2 Answers

When I setup hadoop and spark in my laptop and try to launch spark as "spark-shell --master yarn" I got the same error message.

Solution:

sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml

Add the following property:

<property>       
<name>yarn.nodemanager.vmem-pmem-ratio</name>      
<value>5</value>  
</property>

Then restart hadoop service

stop-all.sh 
start-all.sh
user_68348180
Related