Sqs sensor in airflow not triggering when there is a new message

user_4575909 0
Viewed 2087

I am new to Airflow. I am trying to run my dag whenever there is a message in SQS. I am using SQSSensor to do the same. It triggered for the first run but after that it doesn't invoke when there is a new message.

Please let me know if i am missing something.

default_args = {
    'owner': 'Airflow',
    'start_date': days_ago(2),
    'provide_context': True,
}

dag = DAG('sqs_test', default_args=default_args, schedule_interval='@daily')

task = SQSSensor(
    task_id='sqs_test',
    poke_interval=0,
    timeout=10,
    sqs_queue='https://sqs.us-east-1.amazonaws.com/accountid/test',
    aws_conn_id='aws_default',
    max_message=1,
    dag=dag)
amazon-sqs airflow job-scheduling
1 Answers

it will not invoke when every new message because the schedule interval is set @daily. So it will run next day (as per schedule interval) and then get messages from SQS queue.

thing to note: SQSSensor is just sensor on SQS Que, basically to poll the que. The trigger for DAG is still the schedule we specify in dag definition.

user_139279800
Related