Importing a module in Databricks

Viewed 45

I'm not sure how importing module in python really works in Azure Databricks. First, I was able to make it work using the following:

import sys
sys.path.append("/Workspace/Repos/Github Repo/sparkling-to-databricks/src")
from utils.some_util import *

I was able to use the imported function. But then I restarted the cluster and this would not work even though the path is in sys.path.

I also tried the following:

spark.sparkContext.addPyFile("/Workspace/Repos/Github Repo/my-repo/src/utils/some_util.py")

This did not work either. Can someone please tell me what I'm doing wrong here and suggest a solution. Thanks.

0 Answers
Related