Databricks - push file to GIT

user_6156353 0
Viewed 404

Is there any way to:

  1. Get file (.csv for example) from storage (ADLS)
  2. Push it to GIT (Azure DevOps) repository

using (Azure) Databricks notebook (programatically)?

I tried Databricks Repos, however it works only for notebooks and only via UI and git clone did not work in a notebook.

azure-databricks databricks azure-devops azure
2 Answers

I know databricks repos is quite limited in what git commands you can do (which should be improved in my humble opinion).

Anyway, by default, databricks only 'recognizes' notebook files in your repos. You can allow repos to detect other files as well. Under Settings -> Admin console enable the option Files in Repos. I think you'd need Admin Rights to be allowed to change this setting.

More info: https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html

user_132364210

What I did is:

  1. Copy file from desired location (ADLS) to some temp folder (e.g. FileStore) (using shell or dbutils)
  2. git clone in notebook using shell. cloning using personal access token from Azure DevOps. Files get cloned to driver node so they are not visible nor accessible anywhere (in UI).
  3. Copied wanted files to the cloned repo folder
  4. Pushed back to git (you have to set up user name and email using git config in advance)
user_61563530
Related