How to write/read time stamp from a variable in airflow?

There's an {{ execution_date }} variable in Jinja you can use to get the execution date for the current DAG run.

More info: Airflow - Macros

If you're looking to track something like start time or end time of execution or duration (in seconds) of a particular task instance, that info is stored in the TaskInstance model.

class TaskInstance(Base, LoggingMixin):
    ...
    start_date = Column(UtcDateTime)
    end_date = Column(UtcDateTime)
    duration = Column(Float)

https://github.com/apache/incubator-airflow/blob/4c30d402c4cd57dc56a5d9dcfe642eadc77ec3ba/airflow/models.py#L877-L879

Also, if you wanted to compute the running time of the entire DAG, you could get that from querying the Airflow metadata database around these fields for a particular DAG run.

If you're doing this in your Python code already, you can access the execution_date field on the task instance itself as well instead of using the template layer.

Variables

You can write to and read from Airflow Variables like so:

Variable.set('my_key', 'some value')
my_val = Variable.get('my_key')

You can also perform CRUD operations on variables with the CLI.

Stats

Another thing you might keep in mind if you find yourself working with stats like task duration a lot is Airflow's StatsD integration which gathers metrics on Airflow itself at execution time. You can have these metrics go into a push-based system like StatsD itself, or into a pull-based system like Prometheus / Grafana by using statsd_exporter.

You can do this with the macro execution_date. However be advised that this is a poorly named concept in airflow. It represents the beginning of a scheduled interval period. It will not change within the same dag-run, even if the task is re-run manually. It's there to support idempotent data updates. Which frankly is the optimal way to approach data pipelines. In your case though you said elsewhere that your data fetching api takes a start date and provides all the data up to current, which isn't conducive to being processed idempotently, though you could throw away data after a cut off specified.

So instead you might just take the date after your processing of data has completed, and store that for later. You can store into an Airflow Variable. You might note though that the time you get out of a date command shown below is going to be later then the last time of the data you might have gotten from within your process_data api call for all data from a start date. So it might be better if your processing step outputs the actual last date and time of the data processed as the last line of stdout (which is captured by BashOperator for xcom).

E.G.

from airflow.models import Variable, DAG
from datetime import datetime

def pyop_fun(**context):
  # You could have used execution_date here and in the next operator
  # to make the operator rerun safe.
  # date_string = context['execution_date'].strftime('%Y-%m-%d %H:%M')
  # But elsewhere you said your api is always giving you the up-to-the-minute data.
  # So maybe getting the date from the prior task would work better for you.
  Variable.set(
    'process_last_run',
    context['task_instance'].xcom_pull(task_ids='process_data')

with dag as DAG(…):
  pyop = PythonOperator(
    task_id='set_process_last_run',
    callable=pyop_fun,
    provide_context=True, …)
  shop = BashOperator(
    task_id='process_data',
    bash_command='''
      process_data "{{var.value.process_last_run}}";
      date -u +%Y-%m-%d\ %H:%M''',
    xcom_push=True, …)
  shop >> pyop

# Because the last output line of a BashOperator is pushed into xcom for that
# task id with the default key, it can be pulled by the PythonOperator and 
# stored in a variable.