Running Spark on Kubernetes with Dynamic Allocation

Question

I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.

So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.

I am setting it as below

Master image (spark-master:X)

    FROM 
    ENTRYPOINT ["/opt/spark/sbin/start-master.sh", "-p", "8081", ""]

Worker worker image (spark-worker:X)

    FROM 
    ENTRYPOINT ["/opt/spark/sbin/start-worker.sh", "spark//spark-master:8081" ,"-p", "8081", ""]

Deplyments

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-master-server
spec:
  replicas: 1
  selector:
    matchLabels:
      component: spark-master-server
  template:
    metadata:
      labels:
        component: spark-master-server
    spec:
      containers:
        - name:  spark-master-server
          image: spark-master:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: spark-master

spec:
  type: ClusterIP
  ports:
    - port: 8081
      targetPort: 8081
  selector:
    component: spark-master-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-worker-instance
spec:
  replicas: 3
  selector:
    matchLabels:
      component: spark-worker-instance
  template:
    metadata:
      labels:
        component: spark-worker-instance
    spec:
      containers:
        - name:  spark-worker-server
          image: spark-worker:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081

Questions

Is this setup recommended ?
How can we submit new job from within Kubernetes cluster in absence of yarn (k8s) ?

Reason we are trying not create master and driver dynamically per job (as given in example - http://spark.apache.org/docs/latest/running-on-kubernetes.html) is that it may be an overhead for large no. of small jobs.

user_1305344 · Answer

Is this setup recommended ?

Don't think so.

Dynamic Resource Allocation is a property of a single Spark application "to dynamically adjust the resources your application occupies based on the workload."

Dynamic Resource Allocation spans its resource requirements regardless of available nodes in a cluster. As long as there are resources available and a cluster manager could assign them to a Spark application these resources are free to go.

What you seem to be trying to set up is how to scale the cluster itself up and down. In your case it's Spark Standalone and although technically it's possible with ReplicaSets (just a guess) I've never heard any earlier attempts at it. You're on your own as Spark Standalone does not support it out of the box.

That I think is an overkill since you're building a multi-layer cluster environment: using a cluster manager (Kubernetes) to host another cluster manager (Spark Standalone) for Spark applications. Given Spark on Kubernetes supports Dynamic Allocation out of the box the only worry of yours should simply be how to "throw in" more CPU and memory on demand while resizing Kubernetes cluster. You should rely on the capabilities of Kubernetes to resize itself up and down rather than Spark Standalone on Kubernetes.

user_13985737 · Answer

The spark on k8s operator may provide at least one mechanism to dynamically provision the resources you need to do safe scaling of resources based on demand.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

My thinking is that instead of performing a direct spark submit to a master in a static spark cluster, you could make a call to the k8s API to provision the required instance; or otherwise define these as a cron schedule.

In the dynamic provisioning scenario one thing to consider is how your workloads get distributed across the cluster; we can’t use simple HPA rules for this as it may not be safe to tear down a worker on CPU/Mem levels; spawning a separate cluster for each on demand workload bypasses this but may not be optimal. I would be interested to hear how you get on.

Running Spark on Kubernetes with Dynamic Allocation

2 Answers