Cloud Build builds in worker pools stuck in Queued

user_8055557 0
Viewed 989

We are using Google Cloud Build as CI/CD tool and we use private pools to be able to connect to our database using private IPs.

Since 08/27 our builds using private pools are stuck in Queued and are never executed ou fail due to timeout, they just hang there until we cancel them.

We have already tried without success:

  • Change the worker pool to another region (from southamerica-east1 to us-central1);

  • Recreate the worker pool with different configurations;

  • Recreate all triggers and connections.

Removing the worker pool configuration (running the build in global) executed the build.

cloudbuild.yaml:

steps:
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    id: Backup database
    args: ['gcloud', 'sql', 'backups', 'create', '--instance=${_DATABASE_INSTANCE_NAME}']

  - name: 'node:14.17.4-slim'
    id: Migrate database
    entrypoint: npm
    dir: 'build'
    args: ['...']
    secretEnv: ['DATABASE_URL']

  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    id: Migrate traffic to new version
    dir: 'build'
    entrypoint: bash
    args: ['-c', 'gcloud app services set-traffic ${_SERVICE_NAME} --splits ${_VERSION_NAME}=1']

availableSecrets:
  secretManager:
    - versionName: '${_DATABASE_URL_SECRET}'
      env: 'DATABASE_URL'

options:
  pool:
    name: 'projects/$PROJECT_ID/locations/southamerica-east1/workerPools/<project-id>'

our worker pool configuration:

$ gcloud builds worker-pools describe <worker-pool-id> --region=southamerica-east1 --project=<project-id>

createTime: '2021-08-30T19:35:57.833710523Z'
etag: W/"..."
name: <worker-pool-id>
privatePoolV1Config:
  networkConfig:
    egressOption: PUBLIC_EGRESS
    peeredNetwork: projects/<project-id>/global/networks/default
  workerConfig:
    diskSizeGb: '1000'
    machineType: e2-medium
state: RUNNING
uid: ...
updateTime: '2021-08-30T20:14:13.918712802Z'
google-cloud-build google-cloud-platform
2 Answers

It was my last week discussion with the Cloud Build PM... TL;DR: if you haven't support subscription, or a corporate account, you can't (for now)

In detail, you can check the 1. link of RJC, you will get that enter image description here

If you have a closer look, you can see (with my personal account, even if I have an Organization structure) the Concurrent Builds per worker pool is set to 0. That is the reason of your infinite queue of your build job.

The most annoying part is this one. Click on a Concurrent build per worker pool line checkbox and then click on edit, to change the limit. Here what you get

enter image description here

Read carefully: set a limit between 0 and 0.

Therefore, if you haven't support subscription (like me) you can't use the feature with your personal account. I was able to use it with my corporate account, even if I shouldn't...

For now, I haven't a solution, only this latest message from the PM

The behaviour around quota restrictions in private pools is a recent change that we're still iterating on and appreciate the feedback to make it easier for personal accounts to try out the feature.

user_113725930

The build in queue state can have the following possible reasons:

  1. Concurrency limits. Cloud Build enforces quotas on running builds for various reasons. As a default, Cloud Build has only 10 concurrent build limit, whilst as per Worker Pool, it has a 30 concurrent build limit. You can also further check in this link for the quotas limit.

  2. Using a custom machine size. In addition to the standard machine type, Cloud Build provides four high-CPU virtual machine types to run your builds.

  3. You are using worker pools alpha and has too few nodes available.

Additionally, if the issue still persist, you can submit a bug under Google Cloud. I see that your colleague already submitted a public issue tracker in this link. In addition, if you have a free trial or paid support plan, it would be better to use it to file an issue.

user_165313800
Related