DBT and MWAA integration

Viewed 617

I am trying to integrate DBT with AWS MWAA and i am getting below error:

2021-09-06 07:04:31,290] {{dbt_hook.py:117}} INFO - /usr/local/airflow/.local/bin/dbt run --profiles-dir /usr/local/airflow/dags/dbt/
[2021-09-06 07:04:31,326] {{dbt_hook.py:126}} INFO - Output:
[2021-09-06 07:04:33,280] {{dbt_hook.py:130}} INFO - Running with dbt=0.20.1
[2021-09-06 07:04:35,416] {{dbt_hook.py:130}} INFO - Encountered an error:
[2021-09-06 07:04:35,450] {{dbt_hook.py:130}} INFO - [Errno 30] Read-only file system: 'target/partial_parse.msgpack'
[2021-09-06 07:04:35,636] {{dbt_hook.py:134}} INFO - Command exited with return code 2

My Dag is :

from airflow import DAG
from airflow_dbt.operators.dbt_operator import (
    DbtSeedOperator,
    DbtSnapshotOperator,
    DbtRunOperator,
    DbtTestOperator
)
from airflow.utils.dates import days_ago


default_args = {
    'start_date': days_ago(0)
}

with DAG(dag_id='dbt_snow', default_args=default_args, schedule_interval='@daily') as dag:
    dbt_run = DbtRunOperator(
        task_id='dbt_run',
        dbt_bin='/usr/local/airflow/.local/bin/dbt',
        profiles_dir='/usr/local/airflow/dags/dbt/',
        dir='/usr/local/airflow/dags/dbt/learn/'
    )

I am using the plugin provided :

https://github.com/gocardless/airflow-dbt

https://pypi.org/project/airflow-dbt/

https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production#using-airflow

I tried removing partial parsing by adding config to false and even then i am getting this error.

Any pointer on how to resolve it.

1 Answers

changing the target and logs location to the following made it work:

/usr/local/airflow/dags/dbt/ 

So we need to ship them back, as per the need.

Related