Python - Status of long running process (possibility for multiple processes)

user_989185 0
Viewed 132

I have built an API using Python and FastAPI. With this API, my goal is that I can train custom spaCy NLP models.

This is my code:

'''
main.py
'''
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import subprocess
import sys

app = FastAPI()

class Project(BaseModel):
    token: str

def run_task(cmd, project_path):
    process = subprocess.Popen(cmd, cwd=project_path, shell=True, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    return process


@app.post("/project/train")
async def main(project: Project):
    project_path = '/'.join(["textcat", "projects", project.token]) 
    process = run_task("python -u -m spacy project run all", project_path)
    

    return {
        "status": "success",
        "process_id" : process.pid
    }

I run the app by running:

python -m uvicorn main:app

Then, I trigger a POST request to: http://127.0.0.1:8000/project/train with the below body:

{
   "token":"1234"
}

The subprocess I am triggering python -m spacy project run all is basically running a spaCy workflow, which can take quite some time (hours) to finish.

Lets say that, in my team we are two users, that want to train a custom model each - meaning we will both send a request to the API (with different project tokens), and trigger the training of the NLP model.

Is there a way I can query the running processes to get the output / progress of each?

spacy-3 fastapi spacy python-3.x python
1 Answers

I see 2 ways of doing this:

  • Log your progress in a text file
  • Log your progress in database

In any case you have to write it down somewhere on the disk, and not in memory, so that, in case the application stops unexpectedly, you can still know where your job stopped.

user_61918550
Related