High Concurrency Clusters in Databricks

High Concurrency clusters are intended for use by multiple users, sharing the cluster resources, and isolating every notebook, etc. For all mentioned languages, separate processes are created and they execute the code that is limited to API exposed by Spark to that language. But Scala code will be executed inside the Spark JVM (per machine) that is shared between all users, so you can get access to everything that is inside JVM.

Typically, when using High Concurrency clusters, the Table ACLs are also enabled (only for SQL or SQL+Python), so you can control who can access which data, enforce row/column level access control, etc.

A "high concurrency" cluster is an attempt by Databricks to recreate the performance of normal open source spark (OSS). In normal OSS, multiple applications on a cluster run independently of one another. See https://spark.apache.org/docs/latest/cluster-overview.html

In OSS, the Spark driver logic is hosted in separate/independent processes. Spark applications are co-equals that share the cluster hardware. This is the type of environment that is suitable for a data engineer who wants to maximize the utilization of the available hardware for concurrently running applications.

But Databricks aims to develop a stateful or collaboration experience for "data scientists" who wish to share the cluster and the related notebooks. To accomplish these goals, Databricks has developed a stateful cluster technology via their "driver daemon". It is a single process running on the driver node which coordinates any driver logic that needs to run on the cluster.

When you consider the standard Databricks cluster (ie. an "all-purpose" cluster), you should think of it as a cluster that has the opposite qualities of the "high concurrency" cluster. (see https://docs.microsoft.com/en-us/azure/databricks/clusters/configure)

• High concurrency clusters have "fine-grained sharing". Whereas the default clusters do a lot of intentional resource synchronization (causing temporary conflicts) within the context of the "driver daemon". This is most evident when lots of simultaneous jobs are being submitted.
• High concurrency clusters have "maximum resource utilization". Whereas the default cluster will often be bottlenecked on internal resources within that JVM process ("driver daemon"), and the cluster will rarely use more than one CPU core on the driver node (implying the serialization of all driver logic executing in the cluster).
• High concurrency clusters have "minimum query latencies", whereas latencies on a default cluster become very high - even when there are only 5 or 10 independent jobs running.
• High concurrency clusters "run user code in separate processes", whereas a Databricks cluster will host all driver logic in a single massive JVM process (aka the "driver daemon"). That process becomes a substantial bottleneck for a variety of technical reasons.

In conclusion, the Databricks architecture is built around a single "driver daemon" process (JVM code that was developed primarily with Scala). This can become a bottleneck for certain Spark workloads. But the language bindings for SQL, Python, and R are already designed in a way that executes them outside of the JVM (into a separate "runtime"/"sidecar"). So I'm guessing it was easier to make the adjustments to the architecture that would minimize the concurrency conflicts for those languages when they are already executing in their own sidecars.

In contrast to those languages (SQL, Python and R), the Scala-based logic is assimilated into the "driver daemon" process itself, and doesn't have a separate runtime. A high concurrency mode for Scala could be accomplished, but probably not without the avoidance of that "driver daemon". That would be a massive reversal in their architecture. And it would basically take the platform back to looking like OSS again. They may believe it diminishes their competitive advantages, or it would result in some confusing incompatibilities between their cluster modes.