Long Running App Engine Script "Quitting on terminated signal"

Question

I am attempting to run several long running scripts via app engine. I am a beginner and this is the first project where I have used App Engine and express.

I am handling requests in Node using express.

When I (or a cron job) sends a request to any of my endpoints the script seems to run fine until a random point ~5-10 mins later where I get the following logs:

The Project receives a "/_ah/stop" request
"Quitting on terminated signal" message
"Start program failed: user application failed with exit code -1 (refer to stdout/stderr logs for more detail): signal: terminated"

I cannot work out why this is happening.

My app.yaml:

runtime: nodejs10
instance_class: B2
basic_scaling:
  max_instances: 25
  idle_timeout: 12m

Request handler code:

app.get("/longRunningFunctionOne", async (req, res) => {
    await longRunningFunctionOne();
    res.send("DONE");
});

app.get("/longRunningFunctionTwo", async (req, res) => {
    await longRunningFunctionTwo();
    res.send("DONE");
});

app.get("/_ah/start", async (req, res) => {
    res.sendStatus(200);
});

Absolutely no issues when running locally. Any idea what I am doing to get the premature /_ah/stop request? Because I am using basic scaling I wouldn't get a timeout. Which is described by google as being:

"24 hours for HTTP requests and task queue tasks. If your app doesn't return a request within this time limit, App Engine interrupts the request handler and emits an error for your code to handle."

Any ideas? Perhaps something to do with how I handle /_ah/start which was just a shot in the dark?

user_12265556 · Answer

I figured out that because I send very infrequent requests to the app I was hitting the instance idle timeout before my script had finished/

So even though a task was still running on the app, because it hadn't received a http request in the last ~15 mins, it would send the /_ah/stop request and shutdown the instance.

To keep the instance alive whilst the script is running, I created a function which sends the app a request every min to keep it up and not hit the idle timeout.

I call it the Beegees "stayin' alive" function:

const appUrl = "https://myurl.appspot.com/";

app.get("/script1", async (req, res) => {
    res.send("running script 1");
    stayAlive(script1);
});

app.get("/script2", async (req, res) => {
    res.send("running script 2");
    stayAlive(script2);
});

app.get("/", async (req, res) => {
    res.send(" ");
});

app.listen(process.env.PORT || 4000, () => {
  console.log(`Listening on port ${process.env.PORT || 4000}`);
});

const stayAlive = async (mainAsyncFunc) => {
  const intervalId = setInterval(sendAppRequest, 60000);
  await mainAsyncFunc();
  clearInterval(intervalId);
};

const sendAppRequest = () => {
  console.log("Stayin alive");
  axios.get(appUrl);
};

Seems kinda weird but it works. If you know a better way then please let me know.

Long Running App Engine Script "Quitting on terminated signal"

1 Answers