Identify Composer environment from Cloud function using python

Viewed 47

I have a cloud function that is triggered on file arrival in a cloud bucket. Inside cloud function, a DAG will be called.

If we have any maintenance, composer environment will be down and calling DAG from the cloud function will fail. So In order to avoid this issue, I am planning to maintain one more composer environment, so that one composer will be running even if another one is down.

But I am confused how can we identify in the cloud function, which composer is running so that the DAG can be called. I am using Composer 2.

This is a piece of code extracted from https://cloud.google.com/composer/docs/composer-2/triggering-with-gcf

def make_composer2_web_server_request(url: str, method: str = "GET", **kwargs: Any) -> google.auth.transport.Response:
"""
Make a request to Cloud Composer 2 environment's web server.
Args:
  url: The URL to fetch.
  method: The request method to use ('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT',
    'PATCH', 'DELETE')
  **kwargs: Any of the parameters defined for the request function:
            https://github.com/requests/requests/blob/master/requests/api.py
              If no timeout is provided, it is set to 90 by default.
"""

authed_session = AuthorizedSession(CREDENTIALS)

# Set the default timeout, if missing
if "timeout" not in kwargs:
    kwargs["timeout"] = 90

return authed_session.request(method, url, **kwargs)

Should we make request using the Composer URL like the above from the cloud function to know whether the composer is down or do we have any other methods to attain this?

0 Answers
Related