Spark operator vs spark submit - how to choose which one to use?

user_9574775 0
Viewed 640

There are two ways to run spark application on Kubernetes.

  1. Use spark-submit with k8s:// master URL.
  2. Use spark-on-k8s-operator. With the benefit of CRD in spark operator, we could have declarative management of spark pods using configuration files (YAML).

How do I choose between them in production that running 1000+ spark workloads on Kubernetes?

Any pros and cons of using a spark operator?

apache-spark kubernetes
1 Answers

Pros of spark-on-k8s-operator:

  • creating a k8s deployment of the Spark Operator will spawn pods of SparkApplication type and spawn driver and executor pods.
  • Includes a Prometheus metrics exporter
  • Simplifies application development and management to a YAML file * It can be scheduled using ScheduledSparkApplication

Cons of spark-submit

  • creates naked pods, more here
  • configurations able to be scattered throughout: spark-submit, SessionBuilder, Scala or python code, and k8s.
  • requires additional scheduler

TL;DR - Use spark-on-k8s-operator

user_9448650
Related