I'm writing an API where a user can create a job to be processed with a POST request to /jobs. This returns a 201 with an ID the user can use to check the status of the job later.
While the job is processing, a GET to /jobs/{id} gives back a 204. If the job is complete, it returns a 200 with the results.
I'm unsure what status code to return in the exceptional situation where processing the job fails for some reason. I don't want to return a 204 since that's for jobs that are still going, and not a 404 since the job exists (ie the ID is valid) but it just didn't complete successfully. The closest I've got is a 500, but the API isn't having trouble handling the request, it knows the job has failed. I just don't know how best to communicate that with the status code.