Spark Prometheus metrics name contain random generated data?

user_775988 0
Viewed 175

Metric names that Spark 3.2.0 generates are as follows:

metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_stage_waitingStages_Number{type="gauges"} 0
metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_messageProcessingTime_50thPercentile{type="timers"} 24749.0

I suspect that the part a929f03bcd084dd08ffe5a3c9ba71018 is random generated. How can one write proper Grafana queries to match these metrics? Is it possible to configure Spark not to add these hashes into the metric names? Whare is the purpose for these random hashes in the metrics naming?

(Checked Spark's example spark.metrics.conf for 3.2.0, but found no option to configure this.)

grafana prometheus apache-spark metrics
2 Answers

In Grafana you can filter metrics with the internal __name__ label matcher, and you can write a regular expression to match the above metrics, regardless of the hash. These will match the above metrics:

{\_\_name\_\_=~"metrics_spark_.*_driver_DAGScheduler_stage_waitingStages_Number"}
{\_\_name\_\_=~"metrics_spark_.*_driver_DAGScheduler_messageProcessingTime_50thPercentile"}
user_179223840

In Spark properties, you can specify following:

    "spark.metrics.namespace": "myapp"

So you will obtain:

metrics_myapp_driver_BlockManager_memory_maxOffHeapMem_MB_Number{type="gauges"} 0

Source

user_53462880
Related