Hadoop YARN: How to force a Node to be Marked "LOST" instead of "SHUTDOWN"?

user_749152 0
Viewed 420

I'm troubleshooting YARN application failures that happen when nodes are LOST, so I'm trying to recreate this scenario. But I'm only able to force nodes to be SHUTDOWN instead of LOST. I'm using AWS EMR, and I've tried:

  • logging into a node and doing a shutdown -h now
  • logging into a node and doing sudo stop hadoop-yarn-nodemanager and sudo stop hadoop-hdfs-datanode
  • killing the NodeManager with a kill -9 <pid>

Those result in SHUTDOWN nodes but not LOST nodes.

How do I create a LOST node in AWS EMR?

amazon-emr hadoop-yarn hadoop
2 Answers

NodeManager is LOST means that ResourceManager haven't received heartbeats from it for a duration of nm.liveness-monitor.expiry-interval-ms milliseconds (default is 10 minutes). You may wanna try to block outbound traffic from NM node to RM's IP (or just the port if RM node runs multiple services), but I'm not sure how exactly that can be accomplished in AWS. Maybe use iptables, for example:

iptables -A OUTPUT -p tcp -d <RM's IP> --dport <RM's port> -j DROP
user_6387640

As I suggested in the comments, bringing the interface down on the node induces the node LOST scenario e.g:

ifconfig eth0 down

user_18593700
Related