Retrieve Google Dataflow user defined labels in Google Monitoring & Logging

Viewed 596

I'm running a Dataflow job using the gcloud command with the --parameters labels='{"my_label":"my_value"}' parameter in order to set custom labels (which I can see in the Job Info panel in the UI).

But unfortunately, I'm not able to retrieve this custom label in the Logging stack: I can see the classic Dataflow labels (dataflow.googleapis.com/job_name, dataflow.googleapis.com/job_id, etc.) and the Resource ones (step_id, project_id, region, etc.) but not my my_label custom label.

Is there something I'm missing or is that label propagation I'm expecting does not exist?

Thanks!

EDIT Here is the full command I'm using to run my Dataflow job:

gcloud dataflow flex-template run "my-dataflow-job" \
      --template-file-gcs-location "gs://$(GCP_BUCKET)/my-dataflow-job/templates/my-dataflow-job.json" \
      --region "$(GCP_REGION)" \
      --parameters tempLocation="gs://$(GCP_BUCKET)/my-dataflow-job/temp" \
      --parameters stagingLocation="gs://$(GCP_BUCKET)/my-dataflow-job/staging" \
      --parameters usePublicIps=false \
      --project "$(GCP_PROJECT)" \
      --worker-machine-type="$(DATAFLOW_WORKER_MACHINE_TYPE)" \
      --num-workers=1 \
      --max-workers=1 \
      --parameters labels='{"my_label":"my_value"}'
0 Answers
Related