I'm trying to deploy a NextJs application on App Engine from Cloud Build but is failing because a timeout,
ERROR: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m. but looks like it happends after a crash in an internal lib of App Engine. See the files and loggs below:
app.yaml
env: standard
runtime: nodejs16
service: default
instance_class: F4
inbound_services:
- warmup
automatic_scaling:
min_idle_instances: 1
min_instances: 1
handlers:
- url: /.*
secure: always
script: auto
This cloudbuild file is executed after I push my code into a specific branch in my repo
cloudbuild.yaml
steps:
- name: node:16
id: "npm-install"
entrypoint: npm
args: ["install"]
timeout: 3600s
- name: node:16
id: "env-config"
entrypoint: npm
args: ["run", "create-env"]
env:
...
- name: node:16
id: "build"
entrypoint: npm
args: ["run", "build"]
timeout: 3600s
waitFor: ['env-config']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', './app-develop.yaml', '--verbosity=debug', '--no-cache']
timeout: 3600s # looks like it is ignored by cloud build
waitFor: ['build']
timeout: 3600s # looks like it is ignored by cloud build
options:
machineType: 'E2_HIGHCPU_8'
Logs in the last cloud build step
...
(here a lot of these logs)
...
............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
...............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
................................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
...........................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
.............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] complete. Result: {
"done": true,
"error": {
"code": 4,
"message": "Cloud build did not succeed within 10m.\nAn unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478\nFull build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478"
},
"metadata": {
"@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
"endTime": "2022-09-08T16:07:15.724Z",
"insertTime": "2022-09-08T15:57:07.718Z",
"method": "google.appengine.v1.Versions.CreateVersion",
"target": "apps/guarddog-dev/services/default/versions/20220908t155703",
"user": "479185250478@cloudbuild.gserviceaccount.com"
},
"name": "apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a"
}
failed.
DEBUG: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m.
An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478
Traceback (most recent call last):
File "/builder/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/builder/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
resources = command_instance.Run(args)
File "/builder/google-cloud-sdk/lib/surface/app/deploy.py", line 127, in Run
return deploy_util.RunDeploy(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 692, in RunDeploy
deployer.Deploy(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 471, in Deploy
self.api_client.DeployService(new_version.service, new_version.id,
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 230, in DeployService
return operations_util.WaitForOperation(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 307, in WaitForOperation
completed_operation = waiter.WaitFor(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 261, in WaitFor
operation = PollUntilDone(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 322, in PollUntilDone
operation = retryer.RetryOnResult(
File "/builder/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 249, in RetryOnResult
if not should_retry(result, state):
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
return not poller.IsDone(operation)
File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 182, in IsDone
Take a look that the previous steps just take 1 to 2 minutes
I have to mention that it happends just in our develop environment. When we try to deploy to a production environment the App engine service is updated successfully in a couple of minutes with the same configuration. It sounds like it fails because we are deploying 3 to 5 version into develop environment daily, and many services has been updated and the App Engine get confused/lose or it is because many traffic should be migrated into a new service?
And in desperate attempts we just try many times until it works :`(
