Airflow wrong execution_date for timezone aware DAGs

user_8491129 0
Viewed 782

We're using Airflow v2.2.3 on Kubernetes (KubernetesExecutor), our environment requires a DAG pre-customer, and each customer can be in a different timezone.

Each DAG should be scheduled in its own timezone at midnight, and I saw that it can be achieved using Airflow's timezone aware DAGs

so configuring timezone aware start_date for each DAG worked and made each DAG to execute in its own timezone midnight:

start_date_utc = (datetime.now() - timedelta(days=2)).replace(
        hour=0, minute=0, second=0, microsecond=0)

timezone = pendulum.timezone(get_customer_timezone(customer))
START_DATE = start_date_utc.replace(tzinfo=timezone)

default_args = {
        "owner": "owner",
        "depends_on_past": False,
        "start_date": START_DATE,
}

dag = DAG(
    dag_id,
    schedule_interval="0 0 * * *",
    default_args=default_args,
    tags=[cusotmer_name]
)

date = '{{ execution_date | ds }}'

operator_args = {
    "customer_date": date,
}

My problem is that both the jinja template and dag_run the execution_date (dag_run.logical_date) is still in UTC, and was not adjusted based on the DAGs timezone.

That causes unexpected behavior when running DAGs in different timezones tho execution_date of DAGs with timezone offset earlier than UTC is wrong (2 days before and not 1)

I need some advice please on how can we change the execution_date based on the DAGs timezone

thanks

airflow timezone jinja2
1 Answers

The value of execution_date is in UTC. To convert to different time zone you can do:

{{ execution_date.in_timezone('Europe/Amsterdam') }}

If you set timezone in your DAG you can access it from dag.timezone and use it Jinja.

{{ dag_run.execution_date.astimezone(dag.timezone) }}

You already noticed that execution_date is deprecated macro so you should use logical_date as:

{{ dag_run.logical_date.astimezone(dag.timezone) }}
user_146244090
Related