Custom metrics confusion

user_517781 0
Viewed 889

I added https://micrometer.io to our staging server in google cloud. The metric does not show up in "Cloud Run Revision" resource types. It is only visible if I select "Global" as seen here...

enter image description here

The instructions were very simple and very clear (MUCH UNLIKE opencensus which has a way overdesigned api). In fact, unlike opencensus, it worked out of the box except for it is not recording into "Cloud Run Revision".

I can't even choose the service_name in the filter so once I deploy to production, the metric will be recording BOTH prod and staging which is not what we want.

  1. How do I debug micrometer further
  2. If anyone knows offhand as well what the issue might be, that would be great as well? (though I don't mind learning micrometer and debugging it a bit more).
stackdriver google-cloud-stackdriver micrometer
2 Answers

For now the only available monitored-resource types in your custom metrics are:

  • aws_ec2_instance: Amazon EC2 instance.
  • dataflow_job: Dataflow job.
  • gce_instance: Compute Engine instance.
  • gke_container: GKE container instance.
  • generic_node: User-specified computing node.
  • generic_task: User-defined task.
  • global: Use this resource when no other resource type is suitable. For most use cases, generic_node or generic_task are better choices than global.
  • k8s_cluster: Kubernetes cluster.
  • k8s_container: Kubernetes container.
  • k8s_node: Kubernetes node.
  • k8s_pod: Kubernetes pod.

So, global is the correct monitored-resource type in this case, since there is not a Cloud Run monitored-resource type yet.

To identify better the metrics, you can create metric descriptors, either Auto-creation or manually

user_115364340

For completeness, I have it recording all the JVM stats now but have a new post on aggregation in google's website here that seems to be a new issue...

Google Cloud Metrics and MicroMeter JVM reporting (is this a Micrometer bug or?)

My code that did the trick was (and using revisionName is CRITICALL for not getting errors!!!)

        String projectId = MetadataConfig.getProjectId();
        String service = System.getenv("K_SERVICE");
        String revisionName = System.getenv("K_REVISION");
        String config = System.getenv("K_CONFIGURATION");
        String zone = MetadataConfig.getZone();

        Map<String, String> map = new HashMap<>();
        map.put("namespace", service);
        map.put("job", "nothing");
        map.put("task_id", revisionName);
        map.put("location", zone);

        log.info("project="+projectId+" svc="+service+" r="+revisionName+" config="+config+" zone="+zone);

        StackdriverConfig stackdriverConfig = new OurGoogleConfig(projectId, map);

        //figure out how to put in template better
        MeterRegistry googleRegistry = StackdriverMeterRegistry.builder(stackdriverConfig).build();
        Metrics.addRegistry(googleRegistry);
        //This is what would be used in Development Server
        //Metrics.addRegistry(new SimpleMeterRegistry());
        //How to expose on @backend perhaps at /@metrics

        CompositeMeterRegistry registry = Metrics.globalRegistry;
        new ClassLoaderMetrics().bindTo(registry);
        new JvmMemoryMetrics().bindTo(registry);
        new JvmGcMetrics().bindTo(registry);
        new ProcessorMetrics().bindTo(registry);
        new JvmThreadMetrics().bindTo(registry);

and then the config is simple...

private static class OurGoogleConfig implements StackdriverConfig {

    private String projectId;
    private Map<String, String> resourceLabels;

    public OurGoogleConfig(String projectId, Map<String, String> resourceLabels) {
        this.projectId = projectId;
        this.resourceLabels = resourceLabels;
    }

    @Override
    public String projectId() {
        return projectId;
    }
    @Override
    public String get(String key) {
        return null;
    }
    @Override
    public String resourceType() {
        return "generic_task";
    }

    @Override
    public Map<String, String> resourceLabels() {
        //they call this EVERY time, so save on memory by only passing the same
        //map every time instead of re-creating it...
        return resourceLabels;
    }
};
user_5177810
Related