Exclude Logs from Datadog Ingestion

Question

I have a kubernetes cluster that's running datadog and some microservices. Each microservice makes healthchecks every 5 seconds to make sure the service is up and running. I want to exclude these healthcheck logs from being ingested into Datadog.

I think I need to use log_processing_rules and I've tried that but the healthcheck logs are still making it into the logs section of Datadog. My current Deployment looks like this:

apiVersion: apps/v1
kind: Deployment
[ ... SNIP ... ]
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
        version: "fac8fb13"
      annotations:
        rollme: "IO2ad"
        tags.datadoghq.com/env: development
        tags.datadoghq.com/version: "fac8fb13"
        tags.datadoghq.com/service: my-service
        tags.datadoghq.com/my-service.logs: |
          [{
            "source": my-service,
            "service": my-service,
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]

and the logs coming out of the kubernetes pod:

$ kubectl logs my-service-pod

{
  "@t": "2022-01-07T19:13:05.3134483Z",
  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "@i": "REDACTED",
  "ElapsedMilliseconds": 7.5992,
  "StatusCode": 200,
  "ContentType": "text/plain",
  "ContentLength": null,
  "Protocol": "HTTP/1.1",
  "Method": "GET",
  "Scheme": "http",
  "Host": "10.64.0.80:5000",
  "PathBase": "",
  "Path": "/health",
  "QueryString": "",
  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "EventId": {
    "Id": 2,
    "Name": "RequestFinished"
  },
  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
  "RequestId": "REDACTED",
  "RequestPath": "/health",
  "ConnectionId": "REDACTED",
  "dd_service": "my-service",
  "dd_version": "54aae2b5",
  "dd_env": "development",
  "dd_trace_id": "REDACTED",
  "dd_span_id": "REDACTED"
}

EDIT: Removed 2nd element of the log_processing_rules array above as I've tried with 1 and 2 elements in the rules array.

EDIT2: I've also tried changing log_processing_rules type to INCLUDE at match in an attempt to figure this out:

"log_processing_rules": [
  {
    "type": "include_at_match",
    "name": "testing_include_at_match",
    "pattern": "somepath"
  }
]

and I'm still getting the health logs in Datadog (in theory I should not as /health is not part of the matching pattern)

user_7721999 · Answer

All of these answers are correct in their own ways, but my specific issue was that the Datadog annotations for the source and service were not properly quoted:

        ad.datadoghq.com/my-service.logs: |
          [{
            "source": "my-service",    # Needs Quotes
            "service": "my-service",   # Needs Quotes
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]

user_401096 · Answer

I think the problem is that you're defining multiple patterns; the docs state, If you want to match one or more patterns you must define them in a single expression.

Try somtething like this and see what happens:

"log_processing_rules": [
  {
    "type": "exclude_at_match",
    "name": "exclude_healthcheck_logs",
    "pattern": "\/health|\"RequestPath\": \"\/health\""
  }

user_167920 · Answer

Be sure you are applying the label com.datadoghq.ad.logs to the service container which is generating the log messages... not the Datadog agent container. (That was my mistake.)

Exclude Logs from Datadog Ingestion

3 Answers