If you want to run your job only March, 31st and December, 31st, you can set a cron expression in schedule_interval argument in your DAG definition.

Cron expression will be 0 0 31 3,12 * and can be translated to run at midnight the 31st day of the month for month 3 (March) and month 12 (December). Thus your DAG definition should be:

from airflow import DAG

your_dag = DAG(
    dag_id='your_dag_id',
    ...
    schedule_interval='0 0 31 3,12 *',
    ...
)

For more complicated cases such as run April, 15th and August, 23rd that cannot be defined with cron expression, I guess you should do as Iñigo suggested

You have some options here:

Option 1:

• Create a dag that has as the first step one PytonOperator that the date and fails if it not Dec31 or Mar31.
• Make this first step required to run the next step.

Option 2:

• Create one dag that runs yearly for every date. This looks awful but it can be done easily with a single python file like this:

# Create a dag for an exact date
def createYearlyDagForDate(startdate):

   with DAG(startdate=startdate,
            task_id=f"createdagdordate_{startdate.strftime(month_%m_day_%d)}"
             schedule_interval="@yearly") as dag:

      sparkjob = SparkSubmitOperator(...)

   return dag

for x in [datetime(2021,12,31), datetime(2021,03,31) ]:
   createYearlyDag(x)

The trick here is having a task_id for each dag. If you reuse task_id in the dags, you are overwriting the dag and will have just one declared.