Passing nested dict payloads from `config` to macros in DBT

user_2967377 0
Viewed 837

Problem

I would like to pass a nested dict-of-dicts payload from a model's config to be used in a macro for processing. The payload should be a dictionary with an arbitrary number of keys, and parameters for each of these keys, e.g.:

{{ config(
    payload = {
      'inputs': {
        'input1':  {'param1': false, 'param2': 'test'},
        'input2':  {'param1': false, 'param2': 'test'},
      },
      'other_params': {
        'interval': 1,
        'unit': 'hour',
      }
    })
}}

However I am encountering an error in passing nested dictionary args in the config variable to a macro that uses the payload.

When I try to access the items in the input payload, with e.g. payload.get('inputs'), I get the following error:

Running with dbt=0.20.0
Encountered an error:
Compilation Error in model tf_sample (models/intermediate/development/tf_sample.sql)
  'str object' has no attribute 'get'

  > in macro test_payload (macros/test_payload.sql)
  > called by model tf_sample (models/intermediate/development/tf_sample.sql

However, I believe this issue is specific to how the config variables are being handled, because if I pass the payload dict into the macro directly, the macro will parse successfully.

Question

How can I create a payload dict in this format to be passed to a macro?

Minimal Working Example of the Problem

Below I have created a MWE to demonstrate the issue:

Example macro: loop over the provided payload.get('inputs')

-- macros/test_payload.sql

{% macro test_payload(payload=config.get('payload')) -%}
{% for input, params in payload.get('inputs').items() %}
  -- {{loop.index }}: {{input}}, {{params}}
{% endfor %}
{% endmacro %}
~

However this macro will not succeed to pull in the config.get('payload') in the way that I am expecting:

Example model calling test_payload() -- this will FAIL to parse

The below SQL calls the macro in the way I would like (and would expect to work):

-- model_1.sql --- This will FAIL to parse
{{
  config(
    payload = {
      'inputs': {
        'input1':  {'param1': false, 'param2': 'test'},
        'input2':  {'param1': false, 'param2': 'test'},
      },
      'other_params': {
        'interval': 1,
        'unit': 'hour',
      }
    }
  )
}}
-- this fails
{{ test_payload(config.get('payload')) }}

-- this also fails
{{ test_payload() }}

Example model calling test_payload() with an embedded dict; SUCCEEDS

However if I pass the payload in directly to the macro test_payload, I successfully get the expected output:

-- model_2.sql --- This will SUCCEED! But it doesn't store the payload in the config :(

{{ test_payload(payload={
      'inputs': {
        'input1':  {'param1': false, 'param2': 'test'},
        'input2':  {'param1': false, 'param2': 'test'},
      },
      'other_params': {
        'interval': 1,
        'unit': 'hour',
      }
    }) }}

Result:

  -- 1: input1, {'param1': False, 'param2': 'test'}

  -- 2: input2, {'param1': False, 'param2': 'test'}
dbt
1 Answers

Do you need to have the payload variable inside config for any reason? If that is not the case, you can create variables outside of it and it will work, something like this:

{
  set payload = {
      'inputs': {
        'input1':  {'param1': false, 'param2': 'test'},
        'input2':  {'param1': false, 'param2': 'test'},
      },
      'other_params': {
        'interval': 1,
        'unit': 'hour',
      }
    }
}

{{ test_payload(payload) }}

Then, the macro definition won't need the default config.get('payload'):

{% macro test_payload(payload) -%}
user_70418710
Related