I've a Celery (4.4.7) task which makes a blocking call and may block for long time. I don't want to keep worker busy for long time, so I setup soft_time_limit for the task. My hope was to fail a task (if it's impossible to complete it quickly) and retry it later.
My issue is that SoftTimeLimitExceeded exception is not being raised (I suppose due to the call blocking on OS level). As a result the task is killed by hard time_limit and I don't have a chance to retry it.
@shared_task(
acks_late=True,
ignore_results=True,
soft_time_limit=5,
time_limit=15,
default_retry_delay=1,
retry_kwargs={"max_retries": 10},
retry_backoff=True,
retry_backoff_max=1200, # 20 min
retry_jitter=True,
autoretry_for=(SoftTimeLimitExceeded,),
)
def my_task():
blocking_call_taking_long_time()
What I tried:
- Hard time limit is impossible to intercept
- I expected
ack_latewould push my timed-out task back to the queue, but it doesn't happen.- Tried adding
reject_on_worker_lost, neither value changes things for me
- Tried adding
SoftTimeLimitExceededexception is 100% not there - neitherautoretry_for, nor regulartry ... exceptdon't catch it
For now I ended up with setting explicit timeout for the blocking operation. This requires adding a parameter everywhere along the call chain.
Is there some other path I'm missing?