DBT Merge Partition Prunning

Viewed 43

This is my first time working with dbt! I have successfully implemented an incremental model using #db-bigquery with following config like so

{{ config(
    materialized='incremental', 
    alias='sale_transactions', 
    schema='marts', 
    unique_key='unique_key',
    partition_by={
      "field": "sale_date",
      "data_type": "timestamp",
      "granularity": "day"
    },
    require_partition_filter = false,
    database="iprocure-edw"
) }}

The model runs well, though each time the model runs, it scan the whole destination table. The source data has both new and changed data. The changes are not more than 3 months on the partitioned column.

The merge statement only checks based on unique key DBT_INTERNAL_SOURCE.unique_key = DBT_INTERNAL_DEST.unique_key but has no date filters. I would like to be able to prune out partitions and scan only relevant partitions to avoid full scan which is very costly.

How can one add date filters in the merge query generated by dbt, that will reduce amount of data scanned by BigQuery.

Have found this thought i'm not aware of how/where i can use the solution

0 Answers
Related