I followed the tutorial on https://cloud.google.com/dataproc/docs/concepts/compute/gpus and created a single-node n1-standard-16 Dataproc cluster (base image is: 1.5.35-debian10) and attached Tesla T4 GPU. I installed NVIDIA drivers after creating the cluster and I was able to run my Spark jobs that run on GPU successfully.
However when I stop the master instance, start it again and submit a new Dataproc job, it fails with "Task was not acquired" after 5 minutes, and couldn't find any way to run jobs in the same cluster.
Any help is appreciated.
Edit: After investigating hadoop-yarn logs at /var/log/hadoop-yarn folder as suggested by @Dagang, it seems that it's related to YARN Node Manager. Node manager fails with the following messages.
Edit2: The main failure reason is "Unexpected: Cannot create yarn cgroup Subsystem:devices Mount points:/proc/mounts User:yarn Path:/sys/fs/cgroup/devices/yarn". As suggested in Hadoop official website:, the following lines need to run in the installation script:
chown :yarn -R /sys/fs/cgroup/cpu,cpuacct
chmod g+rwx -R /sys/fs/cgroup/cpu,cpuacct
chown :yarn -R /sys/fs/cgroup/devices
chmod g+rwx -R /sys/fs/cgroup/devices
However these commands already exist in install_gpu_drivers.sh
Complete error log:
Specified path is a directory, use nvidia-smi under the directory, updated path-to-executable
:/usr/bin/nvidia-smi
Trying to discover GPU information ...
=== Gpus in the system ===
Driver Version:460.73.01
ProductName=Tesla T4, MinorNumber=0, TotalMemory=15109MiB, Utilization=0.0%
CGroup controller already mounted at: /sys/fs/cgroup/devices
Initializing mounted controller devices at /sys/fs/cgroup/devices/yarn
Yarn control group does not exist. Creating /sys/fs/cgroup/devices/yarn
Failed to bootstrap configured resource subsystems!
Unexpected: Cannot create yarn cgroup Subsystem:devices Mount points:/proc/mounts User:yarn Path:/sys/fs/cgroup/de
vices/yarn
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl.initializePreMountedCGroupController(CGroupsHandlerImpl.java:424)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl.initializeCGroupController(CGroupsHandlerImpl.java:376)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.gpu.GpuResourceHandlerImpl.bootstrap(GpuResourceHandlerImpl.java:86)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerChain.bootstrap(ResourceHandlerChain.java:58)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:316)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:391)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
2021-09-17 07:18:56,215 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container execu
tor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:393)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
Caused by: java.io.IOException: Failed to bootstrap configured resource subsystems!
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:320)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:391)
... 3 more
2021-09-17 07:18:56,216 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:393)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:878)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:946)
Caused by: java.io.IOException: Failed to bootstrap configured resource subsystems!
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:320)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:391)
... 3 more
2021-09-17 07:18:56,219 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
How to reproduce: I use Google Dataproc Node.js Client library to create Dataproc clusters. Here is my configuration:
request: ClusterCreateInput = {
projectId: 'my-project-id',
region: 'europe-west-1',
cluster: {
projectId: 'my-project-id',
clusterName: 'test-cluster',
config: {
configBucket: 'my-dataproc-log-bucket',
gceClusterConfig: {
zoneUri: 'europe-west-1-d',
serviceAccountScopes: [
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/cloud.useraccounts.readonly',
'https://www.googleapis.com/auth/devstorage.read_write',
'https://www.googleapis.com/auth/logging.write',
],
},
masterConfig: {
numInstances: 1,
machineTypeUri: `n1-standard-16`,
diskConfig: {
bootDiskSizeGb: 100,
bootDiskType: 'pd-standard',
numLocalSsds: 0,
},
accelerators: [
{
acceleratorTypeUri: 'nvidia-tesla-t4',
acceleratorCount: 1,
},
],
imageUri: '1.5.35-debian10',
},
softwareConfig: {
properties: {
'dataproc:dataproc.allow.zero.workers': true,
'spark:spark.executor.instances': '1',
'spark:spark.executor.cores': '16',
'spark:spark.default.parallelism': '16',
'spark:spark.executor.memory': `38000m`,
'spark-env:ARROW_PRE_0_15_IPC_FORMAT': '1',
'spark:spark.executorEnv.LD_PRELOAD': 'libnvblas.so',
},
},
initializationActions: [
{
executableFile: `gs://my-bucket-name/install_gpu_driver.sh`,
},
],
},
},
}
const [clusterOperation] = await this.client.createCluster(request)
const [result] = await clusterOperation.promise()
After creating the cluster, submit a basic job. Stop and start the VM when the job is completed and re-submit a similar job. The cluster will not acquire this new job.