Can I use different machines for Driver and Executor in Spark on Kubernetes? (Azure Kubernetes Service)?
Having two node pools, one for driver using E series machine and another one for Executor using D series machine -> Is this possible?
This is the node pool that I am using currently (Driver and Executor are both Standard_DS15_v2 machines)
Below is my deployment values.yaml file
#
version: 0.0.1
sparkVersion: 3.0.0
image: JAR-name:0.0.1
jar: local:///opt/spark/jars/JAR-name.jar
mainClass: com.company.HelloWorld
fileDependencies: []
environment: test
serviceAccount: my-service-account
imageRegistry: myAcr0.azurecr.io
arguments:
- "Hello World"
driver:
cores: 19
memory: "115g"
executor:
instances: 125
cores: 19
memory: "115g"
hadoopConf:
"fs.abfss.impl": "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem"
"fs.azure.account.key.my_azure_stg_account.dfs.core.windows.net": "KEY_GOES_HERE"
"fs.file.impl": "org.apache.hadoop.fs.LocalFileSystem"
hostNetwork: false
imagePullPolicy: Always
userId: 0
nodeSelector: {"agentpool":"unknown"}
