Receiving HTTP 401 when accessing Cloud Composer's Airflow Rest API

ewertonvsilva's solution worked for me (manually adding the service account to Airflow using gcloud composer environments run <instance-name> --location=<location> users -- create ... )

At first it didn't work but changing the username to accounts.google.com:<service_accounts_uid> made it work.

Sorry for not commenting, not enough reputation.

Based on @Adrien's Bennadji feedback, I'm posting the final answer.

• Create the service accounts with the proper permissions for cloud composer;

• Via gcloud console, add the users in airflow database manually: gcloud composer environments run <instance-name> --location=<location> users -- create --use-random-password --username "accounts.google.com:<service_accounts_uid>" --role Op --email <service-account-username>@<...>.iam.gserviceaccount.com -f Service -l Account

• And then, list the users with: gcloud composer environments run <env_name> --location=<env_loc> users -- list

use: accounts.google.com:<service_accounts_uid> for the username.

Thanks to answers posted by @Adrie Bennadji and @ewertonvsilva, I was able to diagnose the HTTP 401 issue.

The email field in some of Airflow's database tables that are pertaining to users, have a limit of 64 characters (Type: character varying(64)), as noted in: Understanding the Airflow Metadata Database

Coincidentally, my first service account had an email whose character length was just over 64 characters.

When I tried running the command: gcloud composer environments run <instance-name> --location=<location> users -- create --use-random-password --username "accounts.google.com:<service_accounts_uid>" --role Op --email <service-account-username>@<...>.iam.gserviceaccount.com -f Service -l Account as suggested by @ewertonvsilva to add my other service accounts, they failed with the following error: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(64).

As a result, I created new service accounts with shorter emails and these were able to be authenticated automatically. I was also able to add these new service accounts with shorter emails to Airflow manually via the gcloud command and authenticate them. Also, I discovered that the failure to add the user upon first acccess to the REST API was actually logged in Cloud Logging. However, at that time I was not aware of how Cloud Composer handled new users accessing the REST API and the HTTP 401 error was a red herring.

Thus, the solution is to ensure that the combined length of your service account's email is lesser than 64 characters.

Copying my answer from https://stackoverflow.com/a/70217282/9583820

It looks like instead of creating Airflow accounts with

gcloud composer environments run

You can just use GCP service accounts with email length <64 symbols.

It will work automatically under those conditions:

TL'DR version:

In order to make Airflow Stable API work at GCP Composer:

1. Set "api-auth_backend" to "airflow.composer.api.backend.composer_auth"
2. Make sure your service account email length is <64 symbols
3. Make sure your service account has required permissions (Composer User role should be sufficient)

Longread:

We are using Airflow for a while now, and started with version 1.x.x with "experimental" (now deprecated) API's.

To Authorize, we are using "Bearer" token obtained with service account:

# Obtain an OpenID Connect (OIDC) token from metadata server or using service account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)

# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
    method, url,
    headers={'Authorization': 'Bearer {}'.format(
        google_open_id_connect_token)}, **kwargs)

Now we are migrating to Airflow 2.x.x and faced with exact same issue: 403 FORBIDDEN.

Our environment details are:

composer-1.17.3-airflow-2.1.2 (Google Cloud Platform)

"api-auth_backend" is set to "airflow.api.auth.backend.default".

Documentation claims that:

After you set the api-auth_backend configuration option to airflow.api.auth.backend.default, the Airflow web server accepts all API requests without authentication.

However, this does not seem to be true.

In experimental way, we found that if "api-auth_backend" is set to "airflow.composer.api.backend.composer_auth", Stable REST API (Airflow 2.X.X) starting to work.

But there is other caveat to this: for us, some of our service accounts did work, and some did not. The ones that did not work were throwing "401 Unauthorized" error. We figured out that accounts having email length > 64 symbols were throwing error. Same was observed at this answer.

So after setting "api-auth_backend" to "airflow.composer.api.backend.composer_auth" and making sure that our service account email length is <64 symbols - our old code for Airflow 1.x.x started to work for Authentication. Then we needed to make changes (API URLs and response handling) and stable Airflow (2.x.x) API started to work for us in the same way as it was for Airflow 1.x.x.

UPD: this is a defect in Airflow and will be fixed here: https://github.com/apache/airflow/pull/19932