Handling secrets at build time for the purposes of unit & integration testing containers deployed to Cloud Run

Question

Use case

[For docker containers deployed to Cloud Run], run unit tests at build time, some of which require secrets and some level of google authentication. The unit tests might require authentication in order to interact with google services such as cloud storage, or retrieve secret keys from google KMS in order to test interactions with other authenticated APIs. Based on my understanding of gcloud authentication, the only way to provide credentials at build time is by a credentials json file for your service account (whereas when the deployed service is triggered, ADC can use the service account provided by Cloud Run).

Current Approach

Assuming we have generated a credentials file for our service account locally

Check whether the build is initiated locally or by a github trigger. If the build is triggered by a github trigger, only run unit tests that don't require authentication, as I have yet to discover a way to get credentials at build time in this case.
[Proceed only if this is a locally triggered build]. Copy credentials file contents as base64 encoded string into the dockerfile via build argument
Create temporary credentials file during build by decoding the base64 encoded build argument
Run all unit tests using that credentials file
Delete the credentials file as it is not needed in the deployed container

Current Implementation

[Gross and probably unsafe code trigger warning]

$cloud-build-local --config=build_config/cloudbuild_manual.json \
--dryrun=false \
--push \
--substitutions=\
_JSON_CONTENTS="$$(cat $(CREDENTIALS_PATH) | base64)",\
_CONTAINER_ID="$(CONTAINER_ID)" \
.

cloudbuild_manual.json

{
    "name": "gcr.io/cloud-builders/docker",
    "args": [
        "build",
        "--build-arg",
        "JSON_CONTENTS=${_JSON_CONTENTS}",
        "--build-arg",
        "PROJECT_ID=${PROJECT_ID}",
        "--build-arg",
        "RUN_AUTHENTICATED_TESTS=true",
        "-t",
        "gcr.io/${PROJECT_ID}/${_CONTAINER_ID}",
        "."
    ],
    "env":[],
    "timeout": "500s"
}
...

Dockerfile

ARG RUN_AUTHENTICATED_TESTS=false
ENV CREDENTIALS_TEMP_LOCATION credentials_temp.json
ARG JSON_CONTENTS
...
RUN if [ "$RUN_AUTHENTICATED_TESTS" = "true" ] ; then \
    echo "${JSON_CONTENTS}" | base64 -d > $CREDENTIALS_TEMP_LOCATION && \
PYTHONPATH=$PYTHONPATH:`pwd`/ pytest -v --run-auth && \
    rm $CREDENTIALS_TEMP_LOCATION ; \
else \
    PYTHONPATH=$PYTHONPATH:`pwd`/ pytest -v ; \
fi;
...

Problems with this approach

It makes me feel gross inside
Doesn't provide a solution that works for continuous deployment via github triggers
Possibly unsafe?

Possible Solution

Use vcrpy or a similar data mocking library to intercept all unit tested interactions and replace them with recorded versions, thus eliminating the need to have any sort of authentication at build time. Ensure that all tokens and secret keys are censored from the recordings for security (avoiding storing API keys in plaintext in the git repo).

I would love to hear your thoughts on my current approach, alternative approach, and any other solutions you may have. Feel free to trash talk my trash solution :)

user_11372593 · Answer

Nothing is magic. If your Dockerbuild needs authentication parameter, you have to provide it. The env var or the build args are the good way to pass these data to docker build process. You can also avoid this by reusing the file present in the file system.

The secret storage and recovery seems to be a problem. My advice is to store it into Secret Manager. It's the best place for keeping the secret.

Now in your cloudbuild.yaml file, you have to get the secret in a step, and then to pass it at docker build stage

step:
  - name: gcr.io/cloud-builders/gcloud
    entrypoint: bash
    args:
     - "-c"
     - |
         gcloud beta secrets versions access --secret=mySecretName latest > /workspace/mySecret.txt
  - name: gcr.io/cloud-builders/docker
    args: [
        "build",
        "--build-arg",
        "PROJECT_ID=${PROJECT_ID}",
        "--build-arg",
        "RUN_AUTHENTICATED_TESTS=true",
        "-t",
        "gcr.io/${PROJECT_ID}/${_CONTAINER_ID}",
        "."
    ],
    timeout: "500s"

Your Dockerfile

...
ARG RUN_AUTHENTICATED_TESTS=false
ENV CREDENTIALS_TEMP_LOCATION credentials_temp.json
COPY mySecret.txt $CREDENTIALS_TEMP_LOCATION 
...

My opinion: it's safer to NEVER have a credential on your side. Let them securely stored. Secret manager is a great place for this!

user_6481713 · Answer

SOLVED: Instead of finding a workaround to get credentials during build, eliminate the need to have credentials during build via data mocking.

Used VCRPY on a local build to pre-record all HTTP requests used during unit testing.

Used vcr's filter and before_record features to scrub all sensitive data from these recordings, ie.

@pytest.fixture(scope='module')
def vcr_config():
    return {
        # Replace the Authorization request header with "DUMMY" in cassettes
        "filter_headers": [('authorization', 'DUMMY')],
        "before_record_response": my_custom_censor_function(),
        "match_on": ['uri', 'method', 'path']
    }

That fixture works for the following test involving uploading buckets to google storage

def test_authenticated_storage():
    storage_handler = cloud_storage_handler.StorageHandler()
    test_bucket_id = 'bucket-temp-1234-test'
    bucket = storage_handler.create_bucket(test_bucket_id)
    assert(storage_handler.find_bucket(test_bucket_id) is not None)
    storage_handler.delete_bucket(bucket)

NOTE (for anyone trying to use this as a guide): While the VCR solution is much cleaner, I still had to do a slightly janky workaround to satisfy google client libraries' offline functions which are not affected by VCR. In particular, the storage client still needs a credentials.json file, even if we intercept the authentication process afterwards. The trick here was to generate a dummy credentials.json by creating a service account with no permissions, and point the GOOGLE_APPLICATION_CREDENTIALS variable to this test json file during testing.