Azure Databricks Clusters: Can you adjust the number of executors/node?

user_19316040 0
Viewed 256

https://docs.microsoft.com/en-us/azure/databricks/clusters/configure

"Azure Databricks runs one executor per worker node; therefore the terms executor and worker are used interchangeably in the context of the Azure Databricks architecture."

Q1 Does that mean no. of executors per node on azure databricks is fixed to 1?

Q2 Several documentations state 5 cores per executor as a thumb-rule for spark clusters for HDFS throughput? This doesn't seem to be true for Azure databricks.

azure-databricks databricks apache-spark cluster-computing
1 Answers

The Hadoop rules of thumbs aren't applicable for Databricks because in contrast to Hadoop, Databricks doesn't collocate the data with compute, and instead executors are accessing data in cloud storage accounts that have other throughput characteristics compared to the on-prem solutions.

So take as a granted that each node (except driver node) in the cluster is a single executor with number of cores equal to the number of cores on a single machine.

Alex_Ott_186270
Related