Publishing BigQuery Project on github without compromising on key.json

scifi_bot 0
Viewed 39

I have created a bigquery project in GCP and accessed this project using the BQ API in python env (VSCode) and need to submit it as a project to my superiors via github. I have created a service account in GCP and saved its credentials (key.json file) in the python env (VSCode). To reproduce this project, according to my current setup, someone does need a key.json file to access the project. On the other hand, the BQ project in GCP console has a huge dataset uploaded in the cloud which is utilized to make queries etc. So how to even consider making the dataset in google Cloud public (?) As I am new to GCP, I would like to know how to publish such BQ project with proper credentials and no compromise on the private key via .py file in github. Even service account / assigning roles are quite confusing and not as the tutorials from YT.

Any help in this regard will be helpful. Thanks

credentials google-cloud-functions github google-bigquery python
2 Answers

You have several manner to access to GCP resource being authenticated.

  • You can use a service account key file defined directly in your code
  • You can use a service account key file defined in the environment variables
  • You can use the Google Cloud metadata server (available with all Google Cloud services)
  • You can use your own user account

That hierarchy is named ADC (Application Default Credential) and you can leverage it.

In your use case, you aren't on Google Cloud environment, you can remove the metadata server (but it's very useful when you will deploy your code on Google Cloud, keep that in mind)

You want to avoid the service account key file for security reason (and you have absolutely right, avoid using them every time you can, and you can almost every time!)

Last option, use your user account. For that, run the command gcloud auth application-default login. Like that, you create a local credential, based on your Google account. The Google Cloud client libraries in your code know how to leverage that value. (Don't mix up with the gcloud auth login command that is used only for the gcloud CLI, and not by the client libraries; yes not obvious when you are new)

So, you must have the permission to access the ressources (your email). If your boss checkout the code and run it in their environment, they have to be also authenticated and to have access to the resources (don't forget to grant their email on the resources!!)

No credential to share, each environment has their, and it's much more secure without sharing a secret!

guillaume blaquiere0

It seems that the easiest method for your use case is to just add your superior as an IAM user to the Google Cloud Project and use IAM user credentials to avoid service accounts altogether.

To do so:

  1. Add your superior as IAM user under the IAM tab in IAM and Admin.

  2. Grant their IAM user the appropriate IAM roles required to run the application (BigQuery permissions etc.)

  3. Have your superior clone your Github project.

  4. Set ADC credentials by running: gcloud auth application-default login

Note: This step requires the Cloud SDK to be installed, you can also clone the code and run the gcloud command in Cloud Shell as it will have the pre-reqs installed.

  1. Your superior should now be able to run your application without need of a service account.
Jack Wotherspoon0
Related