Migrate Firestore data between Firebase projects

user_88916 0
Viewed 709

I am trying to migrate data from our firebase dev project to our firebase test project using the gcloud cli. To accomplish the task I have created the following script.

#!/bin/bash

SRC_PROJECT=$1
SRC_ACCOUNT="importexport@$SRC_PROJECT.iam.gserviceaccount.com"
SRC_CREDENTIALS=/home/$SRC_PROJECT.json

DEST_PROJECT=$2
DEST_ACCOUNT="importexport@$DEST_PROJECT.iam.gserviceaccount.com"
DEST_CREDENTIALS=/home/$DEST_PROJECT.json

GCLOUD_STORAGE=$3

echo "-------------------------------------------"
echo " Activate Service Account $SRC_PROJECT"
echo "-------------------------------------------"
gcloud --quiet config set project ${SRC_PROJECT}
gcloud beta auth activate-service-account ${SRC_ACCOUNT} --key-file=${SRC_CREDENTIALS} 


echo "-------------------------------------------"
echo " Exporting $SRC_PROJECT"
echo "-------------------------------------------"
gcloud beta firestore export $GCLOUD_STORAGE > ./meta.txt

echo "-------------------------------------------"
echo " Activate Service Account $DEST_PROJECT"
echo "-------------------------------------------"
gcloud --quiet config set project ${DEST_PROJECT}
gcloud beta auth activate-service-account ${DEST_ACCOUNT} --key-file=${DEST_CREDENTIALS} 

echo "-------------------------------------------"
echo "Importing $DEST_PROJECT $(grep -o 'gs://.*$' ./meta.txt)"
echo "-------------------------------------------"
gcloud beta firestore import $(grep -o 'gs://.*$' ./meta.txt)

This script assumes that you have credentials files located in /home that follow the naming convension of <project_id>.json. It further assumes that a service account has been created for both source project and dest project that follows the naming convention of importexport@<project_id>.iam.gserviceaccount.com.

Running the script is as simple as executing the following ...

%> ./migrate.sh dev-project-id test-project-id google-storage

I have ensured that the service accounts both have permissions for the same GCLOUD_STORAGE. And yet I am still getting the following error:

ERROR: (gcloud.beta.firestore.import) PERMISSION_DENIED: The caller does not have permission

Has anyone else tried to accomplish something similar? Any idea why I would still have a permission issue?

Here are the permissions I have assigned to both accounts and both accounts are added as members in each others storage.

  • Cloud Datastore Import Export Admin
  • Cloud Datastore Owner
  • Cloud Filestore Editor
  • Firebase Admin
  • Owner
  • Storage Admin
google-cloud-firestore google-cloud-storage firebase data-migration
1 Answers

When I originally posted this question I had created a dedicated service account designed specifically to have access to all of the environments involved with the transfer. Doing so meant that I had to ensure that the one service account had the right permissions for both Firebase projects as well as the Storage Bucket where the files were stored (in my case this was technically in a 3rd project).

I was never able to get that to work no matter how closely I assigned permissions to match that of the original project accounts.

The solution to the problem was to take one of the default service accounts created to "own" one of the projects, and assign it the required permissions to the other environments. This approach worked where a generic service account failed.

I do not know why that worked, but it is what enabled me to move forward. I hope it works for you as well.

Update There is now new documentation on the Firebase website that shows how to setup the correct permissions for a user to be able to have access to both environments ... BUT there is no way currently to define a service account that spans both environments. Service accounts can't be defined at the company level ... only the project level. This is why I was seeing the failures trying to define one service account with access to two projects.

user_889160
Related