How to create a Distributed spark cluster using Docker

user_9097451 0
Viewed 329

I am trying to create a distributed spark cluster with only one worker using this docker-compose

    master:
  image: gettyimages/spark:2.0.0-hadoop-2.7
  command: bin/spark-class org.apache.spark.deploy.master.Master -h master
  hostname: master
  container_name: spark-master
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_PUBLIC_DNS: <MASTER IP>
  expose:
    - 7001
    - 7002
    - 7003
    - 7004
    - 7005
    - 7077
    - 6066
  ports:
    - 4040:4040
    - 6066:6066
    - 7077:7077
    - 8080:8080
  volumes:
    - ./conf/master:/conf
    - ./data:/tmp/data
    - ~/spark/data/:/spark/data/

worker:
  image: gettyimages/spark:2.0.0-hadoop-2.7
  command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
  hostname: worker
  container_name: spark-worker
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_WORKER_CORES: 2
    SPARK_WORKER_MEMORY: 1g
    SPARK_WORKER_PORT: 8881
    SPARK_WORKER_WEBUI_PORT: 8081
    SPARK_PUBLIC_DNS: <WORKER IP>
  links:
    - master
  expose:
    - 7012
    - 7013
    - 7014
    - 7015
    - 8881
  ports:
    - 8081:8081
  volumes:
    - ./conf/worker:/conf
    - ./data:/tmp/data
    - ~/apps/sparkapp/worker/data:/spark/data/

But the problem is that the docker daemon is creating the containers on the same machine.Which takes away the whole point of having distributed network.How can I create a distributed spark cluster using docker

docker-compose apache-spark docker
1 Answers

If the problem in the same ports for Spark workers, actually you have two options:

  1. Do not expose workers' ports at all - you do not need them for workers to connect to master and work. But may be this is not convenient as you cannot access WebUI of workers
  2. Use special syntax like "8081-8999:8081" so each next worker started with docker-compose up --scale worker=2 will use different port.
user_47909750
Related