Appengine deploy stuck on Updating service [default]

Viewed 49

I'm trying to deploy a NextJs application on App Engine from Cloud Build but is failing because a timeout, ERROR: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m. but looks like it happends after a crash in an internal lib of App Engine. See the files and loggs below:

app.yaml

env: standard
runtime: nodejs16
service: default
instance_class: F4
inbound_services:
  - warmup
automatic_scaling:
  min_idle_instances: 1
  min_instances: 1
handlers:
  - url: /.*
    secure: always
    script: auto

This cloudbuild file is executed after I push my code into a specific branch in my repo

cloudbuild.yaml

steps:
  - name: node:16
    id: "npm-install"
    entrypoint: npm
    args: ["install"]
    timeout: 3600s
  - name: node:16
    id: "env-config"
    entrypoint: npm
    args: ["run", "create-env"]
    env:
      ...
  - name: node:16
    id: "build"
    entrypoint: npm
    args: ["run", "build"]
    timeout: 3600s
    waitFor: ['env-config']
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['app', 'deploy', './app-develop.yaml', '--verbosity=debug', '--no-cache']
    timeout: 3600s # looks like it is ignored by cloud build
    waitFor: ['build']
timeout: 3600s # looks like it is ignored by cloud build
options:
  machineType: 'E2_HIGHCPU_8'

Logs in the last cloud build step

...
(here a lot of these logs)   
...
............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
    ...............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
    ................................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
    ...........................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] not complete. Waiting to retry.
    .............................DEBUG: Operation [apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a] complete. Result: {
        "done": true,
        "error": {
            "code": 4,
            "message": "Cloud build did not succeed within 10m.\nAn unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478\nFull build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478"
        },
        "metadata": {
            "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
            "endTime": "2022-09-08T16:07:15.724Z",
            "insertTime": "2022-09-08T15:57:07.718Z",
            "method": "google.appengine.v1.Versions.CreateVersion",
            "target": "apps/guarddog-dev/services/default/versions/20220908t155703",
            "user": "479185250478@cloudbuild.gserviceaccount.com"
        },
        "name": "apps/guarddog-dev/operations/2493a78d-7c11-4426-a448-e1290b67ca8a"
    }
    failed.    
    DEBUG: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m.
        An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478
        Full build logs: https://console.cloud.google.com/cloud-build/builds;region=asia-east1/e98aaf80-bf4a-4545-822a-4ac67c154413?project=479185250478
        Traceback (most recent call last):
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
            resources = calliope_command.Run(cli=self, args=args)
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
            resources = command_instance.Run(args)
          File "/builder/google-cloud-sdk/lib/surface/app/deploy.py", line 127, in Run
            return deploy_util.RunDeploy(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 692, in RunDeploy
            deployer.Deploy(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 471, in Deploy
            self.api_client.DeployService(new_version.service, new_version.id,
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 230, in DeployService
            return operations_util.WaitForOperation(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 307, in WaitForOperation
            completed_operation = waiter.WaitFor(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 261, in WaitFor
            operation = PollUntilDone(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 322, in PollUntilDone
            operation = retryer.RetryOnResult(
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 249, in RetryOnResult
            if not should_retry(result, state):
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
            return not poller.IsDone(operation)
          File "/builder/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 182, in IsDone

time for each step

Take a look that the previous steps just take 1 to 2 minutes

I have to mention that it happends just in our develop environment. When we try to deploy to a production environment the App engine service is updated successfully in a couple of minutes with the same configuration. It sounds like it fails because we are deploying 3 to 5 version into develop environment daily, and many services has been updated and the App Engine get confused/lose or it is because many traffic should be migrated into a new service?

And in desperate attempts we just try many times until it works :`(

0 Answers
Related