create a dag with airflow

Question

I want to create a dag, which runs every hour, we have a cleanData process which imports and cleans files, followed by a storedata process which sends us a report twice a day at 8am and 6pm. How can I include the times in the dag?

import pandas as pd
import numpy as np

from airflow import DAG
from datetime import datetime
from airflow.operators.python_operator import PythonOperator


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 12, 5),
    'retries': 1
}


def storeData(**context):
    df= context['task_instance'].xcom_pull(task_ids='clean_Data')
    print (df)


def cleanData(**context):
    data = {'Name': ['Tom', 'nick', 'krish', 'jack'],
            'Age': [np.nan, 21, np.nan, 18]}
    df = pd.DataFrame(data)
    df = df.fillna(0)
    return df


dag = DAG(
    'CleaningPipelineDAG',
    default_args=default_args,
    description='Cleaning Data',
    schedule_interval='@once',
)


t1 = PythonOperator(
    task_id='clean_Data',
    provide_context=True,
    python_callable=cleanData,
    dag=dag,
)

t2 = PythonOperator(
    task_id='store_data',
    provide_context=True,
    python_callable=storeData,
    dag=dag,
)

t1 >> t2

user_4856719 · Answer

If the whole DAG can run twice a day (same scheduling for each task, without skipping), then you can use the following schedule_interval:

dag = DAG(
    'CleaningPipelineDAG',
    default_args=default_args,
    description='Cleaning Data',
    schedule_interval='0 8,18 * * *',
)

create a dag with airflow

1 Answers