How can I create custom metrics in Beam for each value of a particular message attribute?

Viewed 385

We have an Apache Beam streaming pipeline that runs on Google Cloud Dataflow. Each message that enters the pipeline is a JSON document and contains an eventType attribute. I would like to monitor the number of messages for each value of eventType. (Ultimately I want to display a chart in Google Cloud Monitoring that displays a stacked bar chart of the tally of each eventType.)

I've found the documentation on creating a counter custom metric in Apache Beam: https://beam.apache.org/documentation/programming-guide/#types-of-metrics which provides this code:

Counter counter = Metrics.counter( "namespace", "counter1");

@ProcessElement
public void processElement(ProcessContext context) {
  // count the elements
  counter.inc();
  //...
}

however that code would simply increment a counter for each message in the pipeline, it doesn't provide a counter per eventType which is what I need. Is there a way to count number of messages per eventType?

0 Answers
Related