What is context variable in Airflow operators

user_17090056 0
Viewed 12698

I'm trying to understand what is this variable called context in Airflow operators. as example:

def execute(self, **context**).

Where it comes from? where can I set it? when and how can I use it inside my function? Another question is What is *context and **context? I saw few examples that uses this variable like this:

def execute(self, *context) / def execute(self, **context).

What is the difference and when should I use *context and **context

airflow python
2 Answers

When Airflow runs a task, it collects several variables and passes these to the context argument on the execute() method. These variables hold information about the current task, you can find the list here: https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html#default-variables.

Information from the context can be used in your task, for example to reference a folder yyyymmdd, where the date is fetched from the variable ds_nodash, a variable in the context:

def do_stuff(**context):
    data_path = f"/path/to/data/{context['ds_nodash']}"
    # write file to data_path...

PythonOperator(task_id="do_stuff", python_callable=do_stuff)

*context and **context are different Python notations for accepting arguments in a function. Google for "args vs kwargs" to find more on this topic. Basically *context accepts non-keyword arguments, while **context takes keyword arguments:

def print_context(*context_nokeywords, **context_keywords):
    print(f"Non keywords args: {context_nokeywords}")
    print(f"Keywords args: {context_keywords}")

print_context("a", "b", "c", a="1", b="2", c="3")

# Non keywords args: ('a', 'b', 'c')
# Keywords args: {'a': '1', 'b': '2', 'c': '3'}
user_30664280

Documentation on the nature of context is pretty sparse at the moment. (There is a long discussion in the Github repo about "making the concept less nebulous".)

In a few places in the documentation it's referred to as a "context dictionary" or even an "execution context dictionary", but never really spelled out what that is.

Apparently, the Templates Reference is considered to be documentation for the context dictionary, although that's not actually mentioned on the page.

As is often the case with Airflow, a look at the source code is sometimes our best bet. One contributor has pointed to the following code block to describe the context dict:

        return {
            'conf': conf,
            'dag': task.dag,
            'dag_run': dag_run,
            'ds': ds,
            'ds_nodash': ds_nodash,
            'execution_date': pendulum.instance(self.execution_date),
            'inlets': task.inlets,
            'macros': macros,
            'next_ds': next_ds,
            'next_ds_nodash': next_ds_nodash,
            'next_execution_date': next_execution_date,
            'outlets': task.outlets,
            'params': params,
            'prev_ds': prev_ds,
            'prev_ds_nodash': prev_ds_nodash,
            'prev_execution_date': prev_execution_date,
            'prev_execution_date_success': lazy_object_proxy.Proxy(
                lambda: self.get_previous_execution_date(state=State.SUCCESS)
            ),
            'prev_start_date_success': lazy_object_proxy.Proxy(
                lambda: self.get_previous_start_date(state=State.SUCCESS)
            ),
            'run_id': run_id,
            'task': task,
            'task_instance': self,
            'task_instance_key_str': ti_key_str,
            'test_mode': self.test_mode,
            'ti': self,
            'tomorrow_ds': tomorrow_ds,
            'tomorrow_ds_nodash': tomorrow_ds_nodash,
            'ts': ts,
            'ts_nodash': ts_nodash,
            'ts_nodash_with_tz': ts_nodash_with_tz,
            'var': {
                'json': VariableJsonAccessor(),
                'value': VariableAccessor(),
            },
            'yesterday_ds': yesterday_ds,
            'yesterday_ds_nodash': yesterday_ds_nodash,
        }

Update

I've since found a mention of the context dictionary in the documentation! If anyone can find any more, I'll be happy to link them here.

When running your callable, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to the context variables you can use in your Jinja templates.

user_20718070
Related