Spark master isn't accessible remotely when cluster is published using docker stack deploy

Viewed 91

I have a Spark cluster running on a remote server, that is set up using a docker-compose.yml on top of bitnami's Spark image. When I spin up the containers using docker-compose up, I can submit jobs to the remote cluster from my machine using the spark://host-ip:port, and everything works fine. However, if I use docker stack deploy to deploy the spark cluster as a swarm service, I get connection refused when I try to submit to the remote cluster.

Here's my partial docker-compose.yml file, which includes the parts that seem relevant to the problem.

version: '3.7'

services:
  spark:
    image: bitnami/spark:3.2.0
    ports:
      - 10001:8080
      - 10000:7077

  spark-worker:
    image: bitnami/spark:3.2.0
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark:7077

The Spark UI port mapping works fine in either method; I can access the Spark UI both when deployed as a stack and when I've started it using docker-compose up. The workers also connect to the master in both scenarios.

By the way, I haven't defined any network in the docker-compose file, and it's using the default network in both cases.

0 Answers
Related