Hadoop on Docker Swarm Problem with accessing DataNodes

user_12510510 0
Viewed 192

I am currently setting up a Hadoop Cluster on four machines. I have one namenode and four datanodes running and they communicate via a docker swarm overlay network.

Now to the problem: When trying to write to HDFS from outside the namenode delegates to the datanodes so the HDFS client tries to access those. But the addresses of the datanodes that the namenode supplies are from the interface of the docker swarm overlay network (in my case 10.0.7.0/24) and for that reason they cannot be reached from outside.

Is there a way to make the namenode return addresses of the datanodes that are reachable from outside? For example use the public IPs of the servers the datanodes are running on?

Thanks in advance!

docker-swarm hdfs hadoop
2 Answers

It seems to be a problem with VIP mode in overlay network (the default) which, apparently, makes the hostname of master node point to another IP and not to the corresponing one due to overlay network behaviour with load balance (source).

Reading this section in networking documentation I have solved the problem changing the endpoint_mode to dnsrr which prevents the routing mesh.

Note that dnsrr mode does not support ingress mode, so you need to specify all the ports in host mode as this section indicates. Keep in mind the following caveat:

If you expect to run multiple service tasks on each node (such as when you have 5 nodes but run 10 replicas), you cannot specify a static target port. Either allow Docker to assign a random high-numbered port (by leaving off the published), or ensure that only a single instance of the service runs on a given node, by using a global service rather than a replicated one, or by using placement constraints.

Thats why you need to add the mode: global setting too when deploy. This is my final docker-compose.yml file ready to run in Docker Swarm:

version: "3.6"
services:
  # Master
  master-node:
    [...] # Extra config
    ports:
      - target: 8088
        published: 8088
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
      [...] # Extra ports
    deploy:
      endpoint_mode: dnsrr
      mode: global # Required by Docker Swarm to make published ports work

  # Workers (here it's not necessary to change the endpoint mode)
  worker:
    [...] # Extra config
user_70583630

I could't solve it using the swarm overlay network so I switched to using the host network. After that, I configured Hadoop to use the public IP of namenode and resourcemanager and it worked!

user_125105100
Related