Query GCP logger for distinct logs based on a field

user_18101942 0
Viewed 1072

Here is a sample log from my project in gcp logger

{
  "insertId": "________",
  "jsonPayload": {
    "stacktrace": "github.com_____",
    "level": "error",
    "msg": "could not update usage",
    "caller": "pkg/_______.go:118",
    "ts": ______.______,
    "requestID": "c7taeaa23akg00e8r0tg",
    "error": "write exception: write errors: [The field 'fieldName' must be an array but is of type null in document {_id: ObjectId('objectID001')}]"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "configuration_name": "configName",
      "service_name": "serviceName",
      "location": "us-central1",
      "project_id": "projectID",
      "revision_name": "revisionName"
    }
  },
  "timestamp": "2022-02-02T15:45:45.867386Z",
  "labels": {
    "instanceId": "____________"
  },
  "logName": "projects/_____/logs/run.googleapis.com%2Fstderr",
  "receiveTimestamp": "2022-02-02T15:45:45.967298989Z"
}

The issue is there are so many of logs with this exact content. My question is that is there query or a set of queries that can be used to retrieve one log with same jsonPayload.error.

For example, if there are 6 logs in which 3 of them have same jsonPayload.error, what I need to achieve is to get 4 logs where the duplicate logs will get cancelled out and only one among them will be at the output along with the other 3 different logs.

google-cloud-platform logging
1 Answers

Interesting question.

Google's Logging query language is a filtering mechanism. Applying a filter reduces the number of entries returned but it does not permit formatting the results to transform the entries.

To transform the results you're gonna to need a bigger boat... I recommend you consider using Google's Cloud SDK command-line tools aka gcloud.

Using this you can filter logs using the queries that you've developed using Log Viewer:

gcloud logging read "${FILTER}" \
--project=${PROJECT}

And (!) you can transform (--format) the results:

gcloud logging read "${FILTER}" \
--format="${FORMAT}" \
--project=${PROJECT}"

NOTE gcloud's formatting does not appear to include unique|distinct functions and so we'll resort to using standard linux (sort|uniq) commands to achieve this.

As an example, a hopefully generic query of cloud.audit.logging operations:

PROJECT="..." # Your Project ID

# You would use "logName=\"projects/${PROJECT}/logs/run.googleapis.com%2Fstderr\""
FILTER="logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Factivity\""

# You would use "value(jsonPayload.error)"
FORMAT="value(operation.producer)"

gcloud logging read  "${FILTER}" \
--project=${PROJECT} \
--format="${FORMAT}" \
--limit=50 \
 > test.log

cat test.log | sort | uniq

Yields:

cloudfunctions.googleapis.com
compute.googleapis.com
container.googleapis.com
k8s.io
servicemanagement.googleapis.com
serviceusage.googleapis.com

NOTE gcloud logging read "${FILTER}" submits the filter to the platform and is run "service-side". The results (which may be large) are then e.g. --format'ted client-side and this can be time/processor-consuming. In the example above, to save repeatedly retrieving the data from the server and then piping it through sort and uniq, it's more efficient to dump the logs into a file first. I've also used --limit to artificially restrict the number of results returned for testing purposes. You may want to use a time filter or something other constraint.

Because you referenced jsonPayload, You can gcloud ... --format=json(...) too in order to extract JSON-formatted logs. As described above, gcloud includes formatting functionality but, also as shown above, sometimes it's easiest to use general-purpose tools. In this case, jq provides powerful ways to transform JSON.

gcloud logging read  "${FILTER}" \
--project=${PROJECT} \
--format="${FORMAT}" \
--limit=50 \
> test.json

cat test.json | jq -r unique

Yields:

[
  null,
  {
    "operation": {
      "producer": "cloudfunctions.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "compute.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "container.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "k8s.io"
    }
  },
  {
    "operation": {
      "producer": "servicemanagement.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "serviceusage.googleapis.com"
    }
  }
]
user_6092900
Related