DataBricks adds # MAGIC in front of every python notebook line

user_5146817 0
Viewed 243

My setup consist git repository connected to Azure DataBricks, this allows to develop databricks python notebooks in PyCharm, push to repo and run updated notebook in DataBricks UI.

Whenever I make change of notebook file from WebUI and do the commit, all lines in notebook file are prepended with # MAGIC:

# Databricks notebook source
%pip install -r "../requirements.txt"

import os

becomes:

# Databricks notebook source
# MAGIC %pip install -r "../requirements.txt"
# MAGIC 
# MAGIC import os

How to fix commits from Azure DataBricks to not change file on push?

pyspark azure-databricks apache-spark python
1 Answers

For future me and others:

above issue related to mixed magic commands and python code. When notebook (from Azure DataBricks UI) is split into separate parts, one containing only magic commands %sh pwd and others only python code, committed file is not messed up.

From text file, separate parts looks as follows:

# Databricks notebook source
# MAGIC %sh pwd
# COMMAND --------

<python code goes here>
user_51468170
Related