Micro meter high memory usage

user_1619503 0
Viewed 928

We are seeing some of our services (Java <> SpringBoot) getting OOM. On checking heap dump, we found the micrometer library is taking 113MB (around 54% of total heap memory).

io.micrometer.statsd.internal.LogbackMetricsSuppressingUnicastProcessor
Jar: io.micrometer:micrometer-core

I did some research online and found creating a lot of distinct tags can lead to this issue. However, this is not the case with our services. We are pushing a lot of metrics to data dog but reusing tags all the time.

micrometer datadog spring-micrometer spring-boot-actuator out-of-memory
1 Answers

Memory issues like this are often caused by using high cardinality tags.

High cardinality here means that the tag can have a large number of unique values. For example, if you want to attach the gender of your users to your metrics, that tag potentially has low cardinality because you might not have a lot of values to choose from. The userID on the other hand is high cardinality since every single user will have a unique one.

Having high cardinality tags can result in a huge amount of meters since you have a meter for every unique value which will result in high memory consumption and your metrics backend will experience similar issues too.

Examples for (potentially) high cardinality tags:

  • ID of the resource, e.g.: userID
  • Email address (basically any user input)
  • Request path instead of template (e.g.: it can contain resourceID)

Also see the Micrometer docs: https://micrometer.io/docs/concepts#_tag_values

Do you have high cardinality tags? If so, you can normalize or remove them.

user_9717350
Related