Run Firestore emulator with Python in GitHub Actions

Question

I have a python package which is using Firestore. I'm trying to write GitHub actions before this I was using Travis now I want to shift on GitHub Actions. Everything is working fine on Travis but on GitHub Actions I'm getting error.

Error: (This error is not issue emulator should have to work without auth becuase it is working in some other repos but not on this)

E: google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

Workflow (not working): Repo FireO Python

name: Python package

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: [3.7]

    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install pytest
          python -m pip install -e .

      - name: Setup NodeJS
        uses: actions/setup-node@v2
        with:
          node-version: "14.x"
      - name: Start Firestore emulator
        run: |
          npm i -g firebase firebase-tools
          firebase setup:emulators:firestore
          firebase emulators:start --only firestore &

      - name: Test with pytest
        run: |
          pytest
        env:
          FIRESTORE_EMULATOR_HOST: localhost:8080

I don't know why the firestore emulator is not working. I have another repo where it is working fine.

Working in another Repo WorkFlow Repo FireO NodeJs

    name: Node.js CI

    on:
    push:
        branches: [main]
    pull_request:

    jobs:
    build:
        runs-on: ubuntu-latest

        strategy:
        matrix:
            node-version: [12.x, 14.x, 15.x]
            # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

        steps:
        - uses: actions/checkout@v2
        - name: Use Node.js ${{ matrix.node-version }}
            uses: actions/setup-node@v1
            with:
            node-version: ${{ matrix.node-version }}

        - name: Install Firestore CLI and start emulators
            run: |
                npm i -g firebase firebase-tools
                firebase setup:emulators:firestore
                firebase emulators:start --only firestore &
        - name: Test Fireo Package
            run: |
                npm ci
                npm test
            env:
                FIRESTORE_EMULATOR_HOST: localhost:8080

Can you please point out what I'm missing what is wrong when I try to test it with python?

To run emulators you don't have to authenticate anything Here in this Repo where it is working without auth but on this Repo where I have python it is not working. I wonder why it is working in some place and not in other

UPDATE

I try to set project id via env GCLOUD_PROJECT="your-project-id" and via command flag --project and it still not working. Because issue is not related to id I think some how python pytest is not able to detect that firestore emulator is running. I don't know why this is happening maybe due to language difference firestore is running on npm and test are running in python Maybe GitHub set both things in different places. Because everything is working on NodeJS Repo because firestore and tests are both running on nodeJS I think somehow we have to sync both of them in one place so they can detect each other.

user_15774183 · Answer

I think you just need to set a project ID, as described in the docs here.

When connecting to the Cloud Firestore emulator from any other environment, you will need to specify a project ID. You can pass a project ID to initializeApp directly or set the GCLOUD_PROJECT environment variable. Note that you do not need to use your real Firebase project ID; the Cloud Firestore emulator will accept any project ID, as long as it has a valid format.

export GCLOUD_PROJECT="your-project-id"

user_9267296 · Answer

I think Akshansha Singhal answered your question already. Please accept his answer if that resolves it for you. You NEED the GCLOUD_PROJECT environment variable when using python. See the docs.

I would like to add a bit though. You are comparing 2 different programming languages. This is really the proverbial apples and oranges.

Actually going through your code a bit, there's an easy explanation. This being fireo-nodejs/Firestore.js. Here you have the project-id in the settings. Going through the python code, there's no setting of the default project-id.

You can either set the GCLOUD_PROJECT env variable outside the python code, or you can add some code to check if it's set inside your python.

Likely you would want to add this in the __init__.py before the from google.cloud import firestore here.

from os import environ

if "GCLOUD_PROJECT" not in environ:
    environ['GCLOUD_PROJECT'] = 'fake-project-id'

user_6769594 · Answer

The previous github actions runs described in the issue were failing because the default credentials could not be found. I've found that the most reliable way to manage setting up the Google Cloud SDK is through setup-gcloud.

To make the SDK setup properly, you'll have to add GCP_PROJECT_ID and GCP_SA_KEY as github secrets. More info can be found on the setup-gcloud README.

Note that GCP_PROJECT_ID has to be a valid project id Also I recommend using Base64 encoding for the JSON content in GCP_SA_KEY. Github scrubs the logs of the secrets info but just to make sure. Thread on this.

Another interesting issue that I was running into is described here. Basically setup-gcloud must be intialized after the python setup or else the python venv won't contain the credentials set by setup-gcloud

Credit goes to isaacna

Run Firestore emulator with Python in GitHub Actions

3 Answers