How to set up dbt with Google Cloud Composer?

user_16106224 0
Viewed 1767

I am trying to install dbt on Google Cloud Composer but run into dependency issues. I have followed the instructions from this article: https://blog.doit-intl.com/setup-dbt-with-cloud-composer-ab702454e27b however at step 2: installing the packages (airflow-dbt & dbt) in composer, it already fails.

I find the following in the Cloud build logs:

ERROR: snowflake-connector-python 2.3.6 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.
ERROR: snowflake-connector-python 2.3.6 has requirement requests<2.24.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: networkx 2.5.1 has requirement decorator<5,>=4.3, but you'll have decorator 5.0.9 which is incompatible.
ERROR: hologram 0.0.13 has requirement jsonschema<3.2,>=3.0, but you'll have jsonschema 3.2.0 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement idna<2.10, but you'll have idna 2.10 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement requests<2.24.0,>=2.18.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: dbt-snowflake 0.19.1 has requirement cryptography<4,>=3.2, but you'll have cryptography 3.0 which is incompatible.
ERROR: dbt-bigquery 0.19.1 has requirement google-api-core<1.24,>=1.16.0, but you'll have google-api-core 1.28.0 which is incompatible.
ERROR: dbt-redshift 0.19.1 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.

My current environment configuration contains: composer-1.13.0-airflow-1.10.12

Has anyone encountered the same problem and have you been able to solve it? I have also tried to install the specific versions of the requirements listed in the logs but this does not resolve the problem.

google-cloud-composer dbt google-cloud-platform
1 Answers

It's a bit painful when you try to install dbt inside the Composer environment. However, there are workarounds for it.

  1. Using external services to run dbt jobs, e.g. Cloud Run.
  2. Using Composer's KubernetesPodOperator. My colleague has put up a nice article on dbt discourse here going through the setup process.
  3. Ignoring Composer's Dependency conflicts by setting Composer's environmental variable IGNORE_PYPI_DEPENDENCY_CONFLICTS to True. However, I don't recommend this as it may cause potential issues.
  4. Creating a Python virtual environment in Composer and install the dbt packages.
user_94655610
Related