I have a Spark cluster running on a remote server, that is set up using a docker-compose.yml on top of bitnami's Spark image.
When I spin up the containers using docker-compose up, I can submit jobs to the remote cluster from my machine using the spark://host-ip:port, and everything works fine. However, if I use docker stack deploy to deploy the spark cluster as a swarm service, I get connection refused when I try to submit to the remote cluster.
Here's my partial docker-compose.yml file, which includes the parts that seem relevant to the problem.
version: '3.7'
services:
spark:
image: bitnami/spark:3.2.0
ports:
- 10001:8080
- 10000:7077
spark-worker:
image: bitnami/spark:3.2.0
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
The Spark UI port mapping works fine in either method; I can access the Spark UI both when deployed as a stack and when I've started it using docker-compose up. The workers also connect to the master in both scenarios.
By the way, I haven't defined any network in the docker-compose file, and it's using the default network in both cases.