How do make it so that every time the incremental load in dbt runs it just updates the new rows from when it last ran?

Viewed 5821

How do make it so that every time the incremental load in dbt runs it just updates the new rows from when it last ran?

{% if is_incremental() %}

    /* code */

    {% endif %}
1 Answers

Your two main resources for doing this are going to be in the dbt docs already:

  1. How to build incremental models in dbt: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models/

  2. Incremental models particular to bigquery: https://docs.getdbt.com/reference/resource-configs/bigquery-configs/#merge-behavior-incremental-models

Most likely the model will look something like:

{{
    config(
        materialized='incremental'
    )
}}

select <columns>
from <my_table>
{% if is_incremental() %}
  where <my_table>.<record_update_timestamp> >= (
      select max(<my_table>.<record_update_timestamp>) from {{ this }}
  )
  {% endif %}

Full example in from docs in: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models/#defining-a-uniqueness-constraint-optional

Related