scheduling events in Python3

user_14001766 0
Viewed 293

I have 4 python scripts that I've mostly run from command-line. I've been trying to schedule them, but so far I haven't found a good way to do this. I have some requirements on how this all should work.

My scripts and what they do:

script number 1; Scans big number of records from database and does some processing.

script number 2; Does more processing, should run only after script number 1 is finished

scripts number 3&4; these scripts are not related to 1 or 2, but they should be run hourly.

Any recommendations what would the best approach for scheduling these scripts in Python?

scheduled-tasks python-3.x python
3 Answers

I understand that the required job needs to schedule. For scheduling jobs, It's good to use CI tools like Jenkins.

  1. make a job for script 1 and 2 and run script 2 after completing script 1.
  2. make two jobs separately for script 3 & 4 that running every hour.
user_55326670

There is the good framework for scheduling python (and not only python!) scripts! Take a look: https://github.com/spotify/luigi It already has 13k stars at github, used in prod at many companies, have a lot of tutorials. It is developed and supported by spotify, so it is actively updated. And, of course, open source :)

Use can install it easily through PyPi next way:

pip install luigi

It even has web-GUI if you prefer that. I really recommend it!

user_49609530

If you want to stick with Python, there's apscheduler. Instead of running the scripts separately, import them and run their functions instead.

Example:

import time
from apscheduler.schedulers.background import BackgroundScheduler
scheduler = BackgroundScheduler()

# your script 1 main function (should be imported instead of creating here)
def my_script1():
    pass

# your script 2 main function (should be imported instead of creating here)
def my_script2():
    pass


# This exists so 
def my_script_runner():
    my_script1()
    my_script2()

# your script 3 main function (should be imported instead of creating here)
def my_script3(id=None):
    print(id)

job_kwargs = {
    'id': 1
}

# Hourly job

scheduler.add_job(my_script_runner, 'interval', [], None, name='myscripts1and2', seconds=7200)

scheduler.add_job(fn, 'interval', [], job_kwargs, name='myscript3', seconds=3600)
scheduler.start()


# Main executer loop
try:
    while True:
        time.sleep(2)
except (KeyboardInterrupt, SystemExit):
    scheduler.shutdown()
user_26354430
Related