Flask-Executor and Flask-SQLAlchemy: Can't get updated data from DB inside of executor function

I'm adding a Flask-based API to my web application to control the start and stop of some network automation functions. I've ran into a strange behavior where functions called by a Flask-Executor .submit() method are seemingly unable to get new or updated data from the database.

I know this question is very involved, so thank you to anyone who shares their time and input. See the end of this question for an overview of my project structure.

The flask-executor documentation says:

When calling submit() or map() Flask-Executor will wrap ThreadPoolExecutor callables with a copy of both the current application context and current request context

I don't quite fully understand what it means by context, but I feel that it might be a good hint about why this should or shouldn't work. (I am using the ThreadPoolExecutor, by the way). I assume that the db SQLAlchemy object is part of the application context, and as such a copy of db should be made available in the executor function. This didn't seem to be the case because I still had to import db in the file containing the function called by the executor, as you'll see later on in this post.

My front end has simple start and stop buttons which send a POST to the following API route:

file: app/api.py

from flask import request
from flask_login import login_required
from app import app, db, executor
from app.models import Project
from datetime import datetime
from automation.Staging import control

@app.route('/api/staging/control', methods=['POST'])
@login_required
def staging_control():
    data = request.json
    project_id = data['project-id']
    action = data['staging-control']

    project = Project.query.get(project_id)
    sp = project.staging_profile
    current_status = sp.status

    if action == 'start':
        if current_status == 'STARTED':
            return {'response': 200, 'message': 'Job already running!'}
        else:
            sp.status = 'STARTED'
            db.session.commit()
            # The executor only spawns the thread if the task status was not already started.
            executor.submit(control.start_staging, project_id)
        
    
    elif action == 'stop':
        if current_status == 'STARTED':
            sp.status = 'STOPPED'
            db.session.commit()

    return {'response' : 200, 'message': 'OK'}

Background

The status of the job is stored in a DB model. If a start action is POSTed, the DB model's status column is updated. Likewise, if a stop action is POSTed, the DB model's status is updated.

The executor's function call to control.start_staging spawns a thread that begins an infinite loop which does some work and then sleeps for X seconds. At the start of each time through the loop, I am trying to check the DB model's status column to determine whether or not to break from the loop and close the thread.

Starting the thread works just fine. The database model gets updated, the executor spawns the thread, and my while loop begins.

Sending the stop action from my frontend works just fine too. The status in the DB is set to STOPPED, and I can see this with manual queries to in my DB shell.

However, the control.start_staging function originally started by the executor still thinks the status is set to STARTED, even though it will actually be updated to STOPPED at some time during the thread's operation. I have attempted to get the updated value as many ways as I can think of from inside the thread. I've seen this similar question.

Here is the control.start_staging function. I've shared a few of the different ways that I've tried to get the updated status in the excerpt below as comments:

file: automation/Staging/control.py

from app import db
from app.models import Project, Staging_Profile
from app.config import STAGING_DURATION_MINS
from datetime import datetime, timedelta
from time import sleep


def start_staging(project_id):    
    project = Project.query.get(project_id)
    print(f"Received START for project {project.project_name}")
    sp = project.staging_profile
    sp.last_activity = datetime.utcnow()
    db.session.commit()
    status = sp.status

    # Staging Loop Start
    while True:

        # This just serves as a force-stop if the job runs for more than STAGING_DURATION_MINUTES minutes.
        if sp.last_activity + timedelta(minutes=STAGING_DURATION_MINS) > datetime.utcnow():
            print(f"Status is: {sp.status}")

            # ATTEMPT 1: does not get updated data
            # status = sp.status

            # ATTEMPT 2: does not get updated data
            # status = Staging_Profile.query.get(project.staging_profile_id).status

            # ATTEMPT 3: does not get updated data
            all_profiles = db.session.query(Staging_Profile).all()
            this_profile = [profile for profile in all_profiles if profile.id == sp.id][0]

            if this_profile.status == 'STOPPED':
                print("Status is STOPPED. Returning")
                break
            
            else:
                print(f"Status is {this_profile.status}")

            # Do work
            do_some_stuff()

        else:
            break
        sleep(5)

    return

Now, what's really puzzling is that I can write data to the database from inside the executor function. The line sp.last_activity = datetime.utcnow() followed by db.session.commit() successfully writes the current time when the thread is started.

My Suspicions

I have built this application in a very modular style approach, and I feel that perhaps this is the source of the issue.

Here is an overview of the relevant parts of my application structure:

app/
├─ __init__.py   # This is where my db & executor are instantiated
├─ api.py        # This is where the /api/staging/control route lives
├─ models.py     # This holds my SQLAlchemy DB classes
├─ routes.py     # This holds my regular front-end routes
├─ config.py     # General config parameters

automation/
├─ Staging/
│  ├─ control.py    # This is where the function passed to the executor is defined
│  ├─ __init__.py   # Empty
├─ __init__.py      # Empty

Thanks again. I will post a resolution or a workaround to this issue when I find one.