Is there any retrieval or transfer charges for moving data from GCS bucket to BigQuery?

Viewed 262

I am doing some POC where I am transferring some Gzip compressed json files to GCS bucket and further consuming those files into BigQuery for analysis. Is there any retrieval or transfer charges for moving data from any GCS bucket(standard or cold storage line) to BigQuery storage.

Also I am using GCS lifecycle configuration on GCS to move my Json files from standard to nearline storage, 30 days after creation. Similarly from nearline to coldline after 60 days and from coldline to archive storage after 90 days. Is there any retrieval charges associated in this case of automatic movement of files among different storage line through lifecycle configuration?

2 Answers

I think the correct answer is here

Unlike changing an object's storage class manually, using SetStorageClass does not rewrite an object. This gives Object Lifecycle Management certain pricing advantages:

  • There are no retrieval fees or early deletion fees associated with the storage class change, even when the object is originally set to Nearline Storage or Coldline Storage.
  • The object's time spent set at the original storage class counts towards any minimum storage duration that applies for the new storage class.

There is no cost for early deletion or retrieval fee when you set another class with lifecycle.


With BigQuery, if you read your data directly from Cloud Storage (federated table), you will pay retrieval fees for the reading with BigQuery (if file in nearline, coldline or archive class).


EDIT 1

BigQuery support native table and federated table.

When you create a native table, you create a load job and the ingestien fee is free. Only steaming ingestion costs on BigQuery (and now the new write API, but still in preview)

With native table, you read the data and copy them to BigQuery storage. Thus you will start to pay for bigQuery storage in addition of Google Cloud Storage storage cost.

But, when you read, you read in BigQuery, and not in GCS. After 90 days without update of the data, they switch automatically to "cold" storage on BigQuery and cost less

For federated table, it's different, you only read the data from GCS when you perform queries.

In summary, both to create the native tables than to read data from federated tables (from GCS), you read the data from GCS to perform these actions. Therefore, if the data are stored in nearline, coldline or archive class in GCS, you will pay for retrieval fees.

Related