I am creating a Dataproc cluster that uses Datalab. To initialize Datalab on the Dataproc cluster, I am creating the cluster from the cloud console and pointing to the datalab.sh initialization script from Google's dataproc-initialization-actions Github repository in the list of initialization scripts. However, when I do this, no Cloud Source Repository is created to version-control the notebooks. I am able to successfully connect to the VM instance and open Datalab, but when I create a new notebook (in the notebooks/ folder), enter some commands, and click Notebook > Save and Checkpoint, nothing happens. When I click the icon to open ungit, ungit says "Nothing to commit" and "No remotes specified":
Is this expected behavior for the Datalab initialization script I am using?
Additional information: The Datalab documentation states that running datalab create VM-instance-name should create a datalabs-notebook Cloud Source Repository. I ran this command for a cluster that was created and then later destroyed, so when I open Cloud Source Repository in the console, I do see a datalabs-notebook source repository. However, when I open that repository from the cloud console, I do not see any notebooks in it.
If I SSH into the VM, and get a shell inside the Docker container running Datalab, I see that a git repository has been created, but there are no remotes:
/content> cd datalab/notebooks
/content/datalab/notebooks> ls
/content/datalab/notebooks> git status .
On branch master
Initial commit
nothing to commit (create/copy files and use "git add" to track)
/content/datalab/notebooks> git remote -v
/content/datalab/notebooks>
(Note, this does not match the contents of the datalab-notebook repository as seen through the Cloud Source Repository - that has a README.md fille that just has the contents # Project Notebooks.)
Do I need to add a datalab create VM-instance-name command to the datalab_init.sh script to create a git repository inside the Datalab container? The Data Engineering training videos I am going through imply this should happen by default.
EDIT/UPDATE 1: To see if this was an issue specific to Dataproc (spoiler: it is), I tried creating a Datalab instance from a Cloud Shell using datalab create my-new-vm. This created a virtual machine running a Datalab instance, and I was able to connect via the browser to this instance. A git repository was correctly created in the notebooks directory, and things generally behaved as I would have expected. This indicates the issue of a Datalab notebooks git repository not initializing specific to the case of the datalab.sh initialization script being run on a Dataproc cluster.
EDIT/UPDATE 2: I tried using ungit to manually create a new git repository in /content/datalabs/notebooks, and then adding the git remote URL for the existing datalab-notebooks repository in Cloud Repository. However, when I do this, ungit requires me to authenticate with the remote repository (enter my username/password). My connection to the ungit page is unsecured HTTP, not HTTPS, and changing the URL to HTTPS results in a "Secure Connection Failed" error. So, I'm not able to test whether I can add the existing datalab-notebooks repository manually (since I am unwilling to send my Google Cloud username/password over an HTTP connection).
EDIT/UPDATE 3: I tried adding the command CLUSTERNAME=/usr/share/google/get_metadata_value name; datalab create ${CLUSTERNAME} to the Dataproc Datalab init script datalab.sh before the dataproc cluster starts up, but this results in datalab: command not found. Not sure why the complete Google Cloud SDK is unavailable on the Dataproc cluster, but that's another question... Meanwhile, the Dataproc team is looking into the diagnostic tarball I sent over.
EDIT/UPDATE 4: Google's Dataproc and Datalab teams have kindly offered to help out and take a look at the issue, so will update this question with any resolution or additional information from them.