I am trying to leverage Google Cloud Functions to run whenever I run an insert on a table within a specific dataset in my GCP project. From what I've seen from other stackoverflow questions, I know it is possible to use Eventarc (2nd Gen) to listen to cloud events and trigger Cloud Functions. From looking at my BigQuery logs, I think what I want is for the Cloud Function to trigger when the logs equal:
protoPayload.methodName:"google.cloud.bigquery.v2.JobService.InsertJob" resource.labels.dataset_id:"the_specific_dataset"
However, after attempting to follow multiple guides, I'm hitting perplexing errors in cloud shell. Sources I've already tried to use: Google Codelabs and This Blogpost.
What I Ran
- in CloudShell, I enabled everything to run it
gcloud config set project org-internal
PROJECT_ID=org-internal
gcloud services enable run.googleapis.com
gcloud services enable eventarc.googleapis.com
gcloud services enable logging.googleapis.com
gcloud services enable cloudbuild.googleapis.com
REGION=us-central1
gcloud config set run/region $REGION
gcloud config set run/platform managed
gcloud config set eventarc/location $REGION- Then I configured my service account in CloudShell to have the roles eventarc.eventReceiver / pubsub publisher
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
gcloud projects add-iam-policy-binding $PROJECT_ID
--member serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com
--role roles/eventarc.eventReceiver
- Then I deployed to CloudRun
SERVICE_NAME=hello gcloud run deploy $SERVICE_NAME \
--image=gcr.io/cloudrun/hello \ --allow-unauthenticated
- I was able to successfully create a trigger for cloud pub/sub. Ran without issue. However, when I tried to apply the event filters specific to table inserts, I ran into issue after issue. Here's what I tried with the errors
gcloud eventarc triggers create $TRIGGER_NAME --destination-run-service=$SERVICE_NAME --destination-run-region=$REGION --event-filters="serviceName=bigquery.googleapis.com" --event-filters="protoPayload.methodName:google.cloud.bigquery.v2.JobService.InsertJob" --event-filters="resource.labels.dataset_id:the_specific_dataset" The error: ERROR: (gcloud.eventarc.triggers.create) argument --event-filters: Bad syntax for dict arg: [protoPayload.methodName:google.cloud.bigquery.v2.JobService.InsertJob].
gcloud eventarc triggers create $TRIGGER_NAME --destination-run-service=$SERVICE_NAME --destination-run-region=$REGION --event-filters="type=google.cloud.audit.log.v1.written” --event-filters="serviceName=bigquery.googleapis.com" --event-filters methodName=google.cloud.bigquery.v2.JobService.InsertJob --event-filters resource.labels.dataset_id=the_specific_datasetThe error: ERROR: (gcloud.eventarc.triggers.create) INVALID_ARGUMENT: The request was invalid: invalid value for attribute 'type' in trigger.event_filters:
I tried a few other ways in formatting (e.g. with quotes and without) but nothing seems to be working. I guess my questions are -- can I filter on "resource.labels.dataset_id" and on "methodName=google.cloud.bigquery.v2.JobService.InsertJob"? If so, what am I doing wrong?