How do I coalesce Stackdriver logs/sinks into one BigQuery project/dataset?

user_2877278 0
Viewed 824

Setting up Stackdriver log sinks to BigQuery is straightforward.

However, I have lots of projects, and instead of each export sink going to its corresponding project, I'd like to coalesce the logs from all my projects to one dedicated project.

The configuration in the Stackdriver sink config doesn't appear to let me select a different project to send the logs to.

enter image description here

How do I select a different project/dataset?

google-cloud-stackdriver google-bigquery google-cloud-platform
3 Answers

You need to select the 'Custom destination' option. This will allow you to plug in a different project/dataset. The default datasets you see are always tied to whichever project you've currently got selected, so you can't select a different project/dataset.

Instead:

  1. Select 'Custom Destination'
  2. Add the destination using this format: bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
  3. Give the sink writer editor permissions on the said project/dataset in step 2.

See more here.

user_28772780

If all of your projects are under a Cloud Folder or under the same organization, you can also create an Aggregated Export as documented in https://cloud.google.com/logging/docs/export/aggregated_exports.

The aggregated sink lives in the folder or org, and exports all logs of children projects to the same destination.

user_10297390

You can configure Bigquery Audit sink at Org level using gcloud command as below

gcloud logging sinks create my-org-bq-sink bigquery.googleapis.com/projects/my-gcp-project/datasets/my_bq_audit_dataset --include-children --organization=MY_ORG_ID

Hope this helps.

user_11583510
Related