How to I use a yaml selector?

user_362202 0
Viewed 1341

I am experimenting with yaml selectors, so far without success. My selectors.yml:

selectors:
  - name: daily
    description: selects models for daily run
    definition:
      exclude:
        - union:
            - "tag:pp_backfill"
            - "tag:inc_transactions_raw_data"
            - "tag:hourly"

And when I try to use it I get an error:

$ dbt ls --selector daily
* Deprecation Warning: dbt v0.17.0 introduces a new config format for the
dbt_project.yml file. Support for the existing version 1 format will be removed
in a future release of dbt. The following packages are currently configured with
config version 1:
 - honey_dbt
 - dbt_utils
 - audit_helper
 - codegen

For upgrading instructions, consult the documentation:
  https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-0-17-0

* Deprecation Warning: The "adapter_macro" macro has been deprecated. Instead,
use the `adapter.dispatch` method to find a macro and call the result.
adapter_macro was called for: dbt_utils.intersect
Encountered an error:
Runtime Error
  Could not find selector named daily, expected one of []

I tried this in both 0.18.1 and 0.19.0, with and without config-version: 2. Any thoughts?

dbt
1 Answers

I think the blocker here might be that you are not currently selecting anything initially to then exclude specific models from using the tag method. Here is solution in my project and then an adoption that might work in your case.

Context

I'm running dbt version 0.19.0 on dbt Cloud. Both of these compiled and successfully ran dbt run --selector daily.

Jaffle Shop Example

stg_customers is tagged with dont_run_me and stg_orders is tagged with also_dont_run_me

selector.yml is the following at the root of the dbt project

selectors:
  - name: daily
    description: selects models for daily run
    definition:
      union:
        - method: path
          value: models
        - exclude:
            - method: tag
              value: dont_run_me
            - method: tag
              value: also_dont_run_me

The logic here is that I'm first select all the models and then exclude the union of the models that have tags dont_run_me and also_dont_run_me.

dbt run --selector daily ended up running everything in my project except stg_customers and stg_orders

Specific Case

If you are trying to select all models except for ones that are tagged as pp_backfill, inc_transactions_raw_data, and hourly, I think the following will do the trick:

selectors:
  - name: daily
    description: selects models for daily run
    definition:
      union: 
        - method: path
          value: models
        - exclude:
            - union:
               - method: tag
                 value: pp_backfill
               - method: tag
                 value: inc_transactions_raw_data
               - method: tag
                 value: hourly
user_112170260
Related