How can we work with multiple DBT projects

user_15581402 0
Viewed 3924

can anyone share how to organise multiple projects in dbt , given the best practices. My present abstract architecture hierarchy is as follows

Analytics 
--.dbt/
-----profiles.yml
--projects/
-----project_1/
----------models/
----------dbt_project.yml
-----project_2/
----------models/
----------dbt_project.yml
--tests/
-----projects/
----------project_1/
----------project_2/

To the point of creating models for either projects_1 or project_2 is working perfectly fine.

but the problem comes when i try to run the test (model unit tests) for project_2.when I run it gives the error ' no dbt_project.yml found at expected path in temp/project_1/ ... ' (the unit tests that i am trying to run is for project_2) however, the paths are absolutely correct but the lookup for dbt_project.yml is in wrong directory (in the temp directory). On the side note, some of the project_2 models does depends on few project_1 models.

can anyone share or reference where i can get the help to solve this sort of multi dbt project issue.

dbt gitlab python database
2 Answers

Like many things, what you're asking here is two sub questions:

  1. Is it possible?
  2. Is it a good idea?

Is it possible?

While you didn't mention what data platform / dialect you are running on, yes, it is possible. Examples and recommendations here:

Is it a good idea?

I think Jeremy breaks out the main arguments for and against really well in this post reply:

But if you've decided that you are already going to do this and don't want a monorepo and you're most interested in the "best" way to implement this, I'd recommend this post by Stefano:

TLDR: Yes, it's possible(ish). Best suggested methods are managing the generate_schema_name / generate_database_name macros OR have low level models defined in packages that your main projects imports.

user_30733400

Can you say more about the specific command you're running when you run dbt test and where you are in this file structure when you run it, as well as the full error message? I'm not sure if I fully understand the problem, but you may want to look into the --profiles-dir option here.

user_76328980
Related