We have multiple GCP projects (dev, acc, pro)that run BigQuery jobs, mainly triggered from Airflow. We notice a rise in our cloud expense for BigQuery slot usage. Is there a BigQuery metadata tables/views where we can see all the queries, data processed, together with their slot usage (if not across projects in our organization, then at least per project). We want to identify which jobs are expensive which we can lessen slot usage.
1 Answers
You have a few options here but to utilize this data as a result of a SQL query you'll want to use the INFORMATION_SCHMEA.JOBS* views. You can either do
JOBS: https://cloud.google.com/bigquery/docs/information-schema-jobs
or
JOBS_BY_ORGANIZATION: https://cloud.google.com/bigquery/docs/information-schema-jobs-by-organization
If you use on-demand pricing then you'll want to focus on the total_bytes_billed column, but for slot usage the best you can get is an average number of slots over the duration of the query (this also has a lot of factors into it including how many other jobs were running concurrently.
You can see an example query for average slot consumption here:
https://cloud.google.com/bigquery/docs/information-schema-jobs#examples
Related