I am using airflow version 2.2.2 and celery as an executor. Here is the thing I am trying to achieve
Previously I have solved this case with celery beats and celery. Since moving the application to the cluster-based scheduler, We are trying with airflow. Can someone suggest we can use airflow for this use case
No. Airflow is not suitable for this use case. Proccess that runs every one second probably don't belog to the scope of batch proccesing.
In your case when combining the two conditions it means also that the task must be completed in 1 second otherwise the pool will build up and more tasks will be scheduled then you can proccess. Airflow is unlikely to be able to handle it.
For this functionality I would check more technologies in the area of event sourcing and streming (assuming the reason you want to run a proccess every 1 second is to respond to data being updated/arived constantly in real time)
