How to tell if my missing logs are due to adaptive sampling in Application Insights?

user_7386097 0
Viewed 925

We use app insights for collecting service telemetry and our services are ASP NET Core running on Service Fabric Clusters. We are observing plenty of missing logs from one of our services and suspect it is due to Adaptive Sampling.

I have a few questions regarding adaptive sampling for better troubleshooting -

  1. Is there a definitive way to know missing logs may be due to adaptive sampling? We are running experiments to turn off adaptive sampling or configure "MaxTelemetryItemsPerSecond" to increase this number but we have no way of verifying these updates have an impact.
  2. And generally what are the symptoms of missing logs due to adaptive sampling? Alternatively, how does one determine adaptive sampling is working for my service?

Any help is appreciated!

azure-application-insights azure
2 Answers

One way to check is to open Logs experience and examine "itemCount" property. If there is no sampling (either not configured or adaptive sampling didn't cross a threshold) then all values should be 1.

One way to think about its value is that this particular event (with itemCount = X) was randomly picked from X events. The greater itemCount value => the data is more sampled.

You can use this query:

union requests, dependencies, traces, exceptions
| where timestamp > ago(24h)
| project itemCount
| summarize count() by itemCount
| order by itemCount

Here is an example of quite heavily sampled application:

enter image description here

user_36469200

You can see it in the Logs pane in your Application Insights instance by running this query:

union requests,dependencies,pageViews,browserTimings,exceptions,traces
| where timestamp > ago(1d)
| summarize RetainedPercentage = 100/avg(itemCount) by bin(timestamp, 1h), itemType

Whenever you see RetainedPercentage less than 100 you are missing log data. More information here.

user_10636180
Related