Apscheduler keeps spawned processes alive for no reason

Viewed 17

EDIT: I found the issue. It was a problem with PyCharm. I ran the .py outside of PyCharm and it worked as expected. In PyCharm I enabled "Emulate terminal in output console" and it now also works there...

Expectations:

  • Apscheduler spawns a thread that checks a website for something.
  • If the something was found (or multiple of it), the thread spawns (multiple) processes to download it/them.
  • After five seconds the next check thread spawns. While the other downloads may continue in the background.

Problem:

  • The spawned processes never stop to exist, which makes other parts of the code (not included) not work, because I need to check if the processes are done etc.

If I use a simple time.sleep(5) instead (see code), it works as expected.
No I cannot set max_instances to 1 because this will stop the scheduled job from running if there is one active download process.

Code:

import datetime
import multiprocessing

from apscheduler.schedulers.background import BackgroundScheduler


class DownloadThread(multiprocessing.Process):
    def __init__(self):
        super().__init__()
        print("Process started")


def main():
    print(multiprocessing.active_children())
        # prints: [<DownloadThread name='DownloadThread-1' pid=3188 parent=7088 started daemon>,
        # <DownloadThread name='DownloadThread-3' pid=12228 parent=7088 started daemon>,
        # <DownloadThread name='DownloadThread-2' pid=13544 parent=7088 started daemon>
        # ...
        # ]

    new_process = DownloadThread()
    new_process.daemon = True
    new_process.start()
    new_process.join()



if __name__ == '__main__':
    sched = BackgroundScheduler()
    sched.add_job(main, 'interval', args=(), seconds=5, max_instances=999, next_run_time=datetime.datetime.now())
    sched.start()

    while True:
        # main()            # works. Processes despawn.
        # time.sleep(5)
        input()
0 Answers
Related