Running Docker container as command breaks API

Question

We have an API that just calls an executable and returns to the client the response from the execution of this command. Its PHP code calling a program via shell_exec. The call usually takes 6-8 seconds to complete. The executable is being substituted by a docker container. In the first step of the process the container will be invoked locally as a program:

shell_exec(docker run --rm containerName");

The problem is that the call to the container makes the client API call (browser) randomly fail with a generic fetch Network Error.

Debugging the issue we have found that the only thing that makes the issue happen is the time the container is running. We have triple checked that there is no timeout anywhere in the chain, and we have verified that the call to the container always finishes with no errors, but at random the client API call just fails.

If you replace the call to the container with a sleep long enough to be greater that a typical container call, and return fake data, the call always success.
If you modify the real call to the container with a very fast call (just ask the containerized program to return its version for example) and then put a sleep in the code and return fake data, the API call always succeeds.
If you call the container as its expected, then sometimes the client throws a fetch Network Error with no more information. In the server logs you can see that the API call returns fine, with the output from the program. Running the app with the browser and the logs open you can see that the browser fails just before the output is logged. In the browser network tab of the developer tools the request is displayed as not ended (no 200, 400 nor 500 HTTP status code).

A brief resume:

The server NEVER fails. All requests are completed and logged just before the response is returned.
The client fails at random but ONLY when the container call is too long. It is not a timeout.
The failed request appears as unfinished in the browser although the server has send the response (seems that too late).

To further debug the issue we have substituted the container for this simple one:

Dockerfile

FROM ubuntu
ADD command.sh /
RUN chmod +x /command.sh
ENTRYPOINT ["/bin/bash", "/command.sh"]

command.sh

sleep 8

With this set up, the client code fails when the sleep is greater than 7 seconds, but succeeds with its below this value. This confirms that the problem is just how long it takes the container to complete.

We have no idea of what can be the source of this problem. The application was running for years with no problem and just substituting the line of code that makes the call starts this behavior.

Is there something in the way we call docker that can make a pending request to be aborted in the client?

Note: I have found no easy way to provide a demo.

Browser unfinished request and logged error:

Error: Network Error
    createError createError.js:16
    handleError xhr.js:84
    dispatchXhrRequest xhr.js:81
    xhrAdapter xhr.js:13
    dispatchRequest dispatchRequest.js:52
    promise callback*request Axios.js:61
    method Axios.js:87
    wrap bind.js:9

Running Docker container as command breaks API

0 Answers