What is the Python API I can use to calculate the cost of a BigQuery query?

Viewed 1558

Did any one know the how to use the Python Google Cloud BigQuery API to calculate the cost of a query?

3 Answers

1. Get the amount of data processed by your query

Using the Google Cloud Python SDK, you can set the dry_run flag.

job_config = bigquery.QueryJobConfig()
job_config.dry_run = True
job_config.use_query_cache = False
query_job = client.query(
  ...
)

2. Calculate cost from total_bytes_processed

If you're using on-demand pricing look at query_job.total_bytes_processed to see how much processed data you'd be billed for. Current cost (as of October 2019) is $5 per TB processed, where the first TB is free.

So, you can get the cost with something like this:

cost_dollars = (query_job.total_bytes_processed / 1024 ** 4) * 5

I don't think you can calculate the cost directly, but you can use the dryRun parameter in the python API to get the number of bytes processed, which you can then convert to cost (at the listed on demand price of $5/TB).

Something like:

from google.cloud import bigquery

job_config = bigquery.QueryJobConfig()
job_config.dry_run = True
job_config.use_query_cache = False
query_job = bigquery.Client().query(
    (
        "SELECT my_column FROM my_table"
    ),
    location="US", # or wherever your data is
    job_config=job_config,
)

print("{} bytes will be processed".format(query_job.total_bytes_processed))

Here's a reference to the cost estimation docs as well.

If you have a project in place, you might want to do :

from google.cloud import bigquery

bqclient = bigquery.Client(
    project='myproject',
)

query = "SQL BLA BLA SELECT etc.. " 
job_config = bigquery.QueryJobConfig()
job_config.dry_run = True
job_config.use_query_cache = False
query_job = bqclient.query(
    
    (
       query
    ),
    location="US", # or wherever your data is
    job_config=job_config,
)

cost_dollars = (query_job.total_bytes_processed / 1024 ** 4) * 5

print(f"{query_job.total_bytes_processed} bytes will be processed , cost ~{cost_dollars}$")
Related