Export BigQuery logging: which resource types to select (and what is the difference between them?)

user_3489155 0
Viewed 344

I want to export relevant BigQuery (audit) logging to a BQ dataset.

So I have to create a log sink for that.

I found a blog that sets up the BQ logging only using bigquery_project or bigquery_dataset:
https://dataform.co/blog/exporting-bigquery-usage-logs

I noticed that there are several more resource.types to query on:

  • bigquery_project
  • bigquery_resource
  • bigquery_dataset
  • bigquery_table
  • bigquery_dts_config
  • bigquery_dts_run
  • bigquery_biengine_model

I found that list here:
https://cloud.google.com/logging/docs/api/v2/resource-list#resource-types

So the last three have to do with data transfer and biengine.

But I wonder what is the difference between project, resource and dataset?

What info is in those 3 different kinds of logs? And: which of the three (or which of the seven resource types) should I use?

log sink for exporting BQ logs

google-cloud-logging google-bigquery google-cloud-platform
1 Answers

AuditData payload will return resource.type set to bigquery_resource, not bigquery_dataset. In BigQueryAuditMetadata messages, resource.type is set to one of the following values: bigquery_dataset for operations to datasets such as google.cloud.bigquery.v2.DatasetService.* such as resource.labels.dataset_id contains the encapsulating dataset and bigquery_project for all other called methods, such as jobs resource.labels.location contains the location of the job.

resource.type="bigquery_project" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a project.

resource.type="bigquery_dataset" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a dataset.

resource.type="bigquery_dts_config" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a Data Transfer Service configuration.

resource.type="bigquery_dts_run" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for a Data Transfer Service.

resource.type="bigquery_biengine_model" AND logName:"cloudaudit.googleapis.com" is used to get BigQuery audit logs for BI Engine Model.

Example:

To show log entries from a given transfer config_id, in the Query builder, add the following filter:

resource.type="bigquery_dts_config"
labels.run_id="transfer_config_id"

For more information you can refer to this document.

user_157474140
Related