How to change yarn-site.xml properties for worker nodes in my google dataproc cluster via init action script?

Viewed 179

I want to create a dataproc cluster having 1 master node and 2 worker nodes but yarn-site property "yarn.timeline-service.reader.webapp.https.address" should have different values for each worker nodes and master node.

I am trying the below method but worker 0 is having the same value as worker 1:

readonly role=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
readonly WORKER_COUNT=$(/usr/share/google/get_metadata_value attributes/dataproc-worker-count)
readonly DATAPROC_CLUSTER=$(/usr/share/google/get_metadata_value attributes/dataproc-cluster-name)
function yarn-site_update-workers {
    if [[ "${role}" != "Master" ]]; then
    for ((c = 0; c < ${WORKER_COUNT}; c++)); do
    cat <<EOF >yarn-site.xml
    <?xml version="1.0"?>
    <configuration>
      <property>
        <name>yarn.timeline-service.reader.webapp.https.address</name>
        <value>https://${DATAPROC_CLUSTER}-w-${c}.c.${PROJECT_ID}.internal:8192</value>
      </property>
    </configuration>
    EOF
      bdconfig merge_configurations \
        --configuration_file "${HADOOP_CONF_DIR}/yarn-site.xml" \
        --source_configuration_file yarn-site.xml \
        --clobber
    done    
    fi
}
1 Answers

In this case, you don't want to use for loop, instead you want to get the hostname of the worker node. Replacing ${DATAPROC_CLUSTER}-w-${c} with ${HOSTNAME} will do the job.

BTW, I don't quite understand why you have to use hostname? Can you simply use 0.0.0.0?

Related