How do I put all my bigquery queries inot a repository or is that even a thing

user_1163960 0
Viewed 582

I have a project that is a bunch of bigquery tables and queries and schedules queries etc. Since its my only copy in the whole wide world of this code and it took me a long time to stumble across it all (thank you stackoverflow), how do I put all those sqls into a repository and save it, or do people only do that with real code like programs and such.

I did this so far: https://cloud.google.com/source-repositories/docs/quickstart#before-you-begin https://cloud.google.com/source-repositories/docs/authentication https://chrome.google.com/webstore/detail/save-bigquery-to-github/heeofgmoliomldhglhlgeikfklacpnad

I made a local repository with git for windows. I made a respository in google and connected my project to it. The repository is currently empty and there isn't a big button anywhere that I can click to just copy the sqls into the repository. There are quickstarts to make code locally and flow it all the way up to google, but I want the code going in the opposite direction. You'd think they'd make it this easy, but I think they want it to be inaccessible to noobs. It seems like it's just easier to copy and paste the sqls one at a time to notepad++ like I would postgres.

google-bigquery repository
1 Answers

The best solution for that is to use a managing tool like DBT or Dataform.

In this kind of tool, you store your infrastructure information in YAML files (dataset organisation, what tables you have, their schema, their materialization -view, table, materialized view, etc), and your queries, UDFs, functions, etc in files with a .sql extension.

(Note you can use templating which is awesome).

Once your queries are in place, you can build your project with simple commands (for dbt for instance: dbt run). There is a lot of additional functionalities that are extremely useful (ex: data lineage, documentation website).

And since you are manipulating files, you then only need to git add git commit git push to your repository to safely store your whole project.

Maybe you can have a look first at Dataform: it has been acquired by Google earlier this year, and it will probably be integrated in BigQuery in the future (becoming the de facto recommended way to manage your data pipelines).

user_150626050
Related