Error in Cloud composer with data build tool (dbt) path ['name']: 'jaffle_shop' does not match '^[^\\d\\W]\\w*$'

user_16917271 0
Viewed 287

I am testing a deployment of dbt within Cloud composer. On my local machine (Ubuntu 20.04) I have got success in running the dbt models with airflow. When running on Google Cloud composer I get the following error

{subprocess.py:74} INFO - Output:
{subprocess.py:78} INFO - Running with dbt=0.21.0
{subprocess.py:78} INFO - Encountered an error while reading the project:
{subprocess.py:78} INFO -   ERROR: Runtime Error
{subprocess.py:78} INFO -   at path ['name']: 'jaffle_shop' does not match '^[^\\d\\W]\\w*$'
{subprocess.py:78} INFO - 
{subprocess.py:78} INFO - Error encountered in /home/airflow/gcs/dags/dbt_project.yml
{subprocess.py:78} INFO - Encountered an error:
{subprocess.py:78} INFO - Runtime Error
{subprocess.py:78} INFO -   Could not run dbt
{subprocess.py:82} INFO - Command exited with return code 2
{taskinstance.py:1503} ERROR - Task failed with exception

We are using a BashOperator to run dbt models in Airflow. Initially had some problems with dependencies but they were solved. Using a standard dbt_project.yml file with a single model just to test how this works. Another way is to use Docker but we need try if this works.

Edit

Versions

dbt: 0.21.0

cloud-composer: 1.17.1

airflow: 2.1.2

Pypi Packages airflow-dbt: 0.4.0

dbt: 0.21.0

jsonschema: 3.1 (Added this as Pypi was throwing an error about the version

I really appreciate if anyone can help

Pete

google-cloud-composer dbt airflow
1 Answers

The problem here is the jsonschema dependency. Version 3.1.0 does not work, while versions 3.1.1 and 3.2.0 will work--and should also work within Composer's dependency requirements.

There looks to have been an issue with switching to js-regex for the jsonschema folks in 3.1.0, which caused them to revert back to regular re in 3.1.1.

There are some details here, and a couple of related issues described here and here.

In general, it would be much nicer if Cloud Composer supported virtual environments to avoid this entire dependency-collision mess, but apparently Google does not support that approach.

user_19384980
Related