Build a container image from inside a cloud function

user_14272242 0
Viewed 178

Context: I am training a very similar model per bigquery dataset in Google Vertex AI, but I want to have a custom training image for each existing dataset (in Google BigQuery). In that sense, I need to programatically build a custom Docker Image in the container registry on demand. My idea was to have a Google Cloud Function do it, being triggered by PubSub topic with information regarding which dataset I want to build the training container for. So naturally, the function will write the Dockerfile and pertinent scripts to a /tmp folder within Cloud Functions (the only writable place as per my knowledge). However, when I try to actually build the container within this script, apparently, it doesn't find the /tmp folder or its contents, even though they are there (checked with logging operations).

The troubling code so far:

def build_container(dataset=str):

    with open('container_template/Dockerfile','r') as f:
        dockerfile = f.read()
    dockerfile = dockerfile.replace('@dataset',dataset)
    f.close()

    os.makedirs(os.path.dirname('/tmp/script-location'), exist_ok=True)

    with open('/tmp/Dockerfile','w') as docker_config:
        docker_config.write(dockerfile)

    docker_config.close()

    shutil.copy('container_template/script-location/script.py','/tmp/script-location/script.py')

    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','/tmp']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

    logger.info('Build Result: {}'.format(build_result.status))

When I check the cloud build logs I get: Step #0: unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/Dockerfile: no such file or directory

google-artifact-registry google-cloud-build google-cloud-functions docker python
1 Answers

I've locally tested building a container image using Cloud Build Client Python library. It turns out to have the same error even the Dockerfile file is existing in current directory:

error:

Step #0: unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /workspace/Dockerfile: no such file or directory

build steps:

    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','.']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

Since it uses the API method I followed this documentation. You will see the source is present in API method. It is the missing key to move forward the problem. In StorageSource, you must to specify the bucket and object_. You need to compress your source code and upload it in Cloud Storage bucket. For example:

  1. Run the following command to compress your source code:
tar -cvzf sourcecode.tar.gz .
  1. Upload to Cloud Storage bucket(you can use Cloud Build bucket):
gsutil cp sourcecode.tar.gz gs://myproject_cloudbuild
  1. build.source:
    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()
    
    build.source = {"storage_source":{"bucket":"myproject_cloudbuild", "object_":"gs://myproject_cloudbuild/sourcecode.tar.gz"}}

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','.']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

As a result it solved the problem building an image using Client Library. I recommended to do all of this inside your Cloud Function.

user_87539910
Related