How to make a reliable Celery task with a blocking call inside?

Viewed 12

I've a Celery (4.4.7) task which makes a blocking call and may block for long time. I don't want to keep worker busy for long time, so I setup soft_time_limit for the task. My hope was to fail a task (if it's impossible to complete it quickly) and retry it later.

My issue is that SoftTimeLimitExceeded exception is not being raised (I suppose due to the call blocking on OS level). As a result the task is killed by hard time_limit and I don't have a chance to retry it.

@shared_task(
    acks_late=True,
    ignore_results=True,
    soft_time_limit=5,
    time_limit=15,
    default_retry_delay=1,
    retry_kwargs={"max_retries": 10},
    retry_backoff=True,
    retry_backoff_max=1200,  # 20 min
    retry_jitter=True,
    autoretry_for=(SoftTimeLimitExceeded,),
)
def my_task():
    blocking_call_taking_long_time()

What I tried:

  • Hard time limit is impossible to intercept
  • I expected ack_late would push my timed-out task back to the queue, but it doesn't happen.
    • Tried adding reject_on_worker_lost, neither value changes things for me
  • SoftTimeLimitExceeded exception is 100% not there - neither autoretry_for, nor regular try ... except don't catch it

For now I ended up with setting explicit timeout for the blocking operation. This requires adding a parameter everywhere along the call chain.

Is there some other path I'm missing?

0 Answers
Related