@monthly cron job is not reliable

user_633961 0
Viewed 1455

Our customer wants us to create a report every month.

In the past, we used a @monthly cron job for this task.

But this is not reliable:

  1. The server could be down in this minute. Cron does not re-run those jobs
  2. If the server is up, the database could be unreachable in this moment.
  3. If the server is up and DB is up, there could be a third party system which is not reachable
  4. There could be a software bug.

What can I do, to be sure that the report gets created monthly?

It is a Django based web application

cron django python
5 Answers

Use a decent scheduler

celery beat is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster.

You create a periodic task with the report function job. If the job fails celery will retry following the retry policy that you have set.

Celery doc - Periodic Tasks

user_72647100

Too many moving pieces and consequently options to consider. But problem 1 means you need some form of external way to track success (otherwise one option would have been for your server task - say a bash script - to keep retrying N times and sleeping in between retries, until the report generation is successful).

If you want a full-blown solution you can use for many different future needs, you can look into the available third party schedulers like Jenkins or SOS Berlin.

If you are looking for a simpler solution, you can schedule the report script via cron to run many times (say every hour for a few days at the end of the month), then have it keep track of whether the report was generated and sent successfully (this could be as simple as creating a file and checking for its existence, or writing a value to a database).

user_40772470

You should write some script that will test conditions and perform all required operations.

if is_work_finished_less_then_month_ago():
    return
else:
    try:
        generate_normal_report()
    except some_error as e:
        report_about_error(e)

Then run it every hour or day.

If you afraid of too many error_reports then do the same thing in report_about_error() method: check last time you sent report and do not send it if it's too often.

user_20394710

The easy way is that you can write an helper such that the script checks for the availability of the report first and generates the report only if the report is unavailable.

Then schedule the script to run every hour on that day of the month.

change your script as follows:

if [[ -f <report_name> ]]
then
    echo "report exists" 
    exit 1
else
    echo "run generate report script"

crontab entry (To run every hour on 28th of every month):

0 0-23 28 * * <name_of_helper_script>
user_50895700

cron is not able to do what you're asking for. And kicking off cron-like tasks via your Django app is not what Django was created for, and will only work if you handle all the edge cases and make sure to pick up missed runs if your app is malfunctioning, etc. It will be a rabbit-hole of error handling, state management and concurrency considerations.

I would suggest one of two options:

  • Kick off the job with cron anyway, but write a Nagios check to ensure that you're notified if the report wasn't generated for some reason (and handle those rare cases manually).
  • Use Airflow or any other framework that was created specifically for the purpose of handling scheduling, monitoring, retrying and logging of scheduled tasks.

The former is what you want if this is really a one-off job. The latter is better if you have more tasks like this coming your way.

user_2196400
Related