Databricks: cannot import custom python module

Question

I am trying to create a python package, which then I want to install on my databricks cluster. I have tried different approaches, but they all end up with ModuleNotFoundError error. I want to create it out of this data_aggregation directory, which is marked as package, according to my pycharm. It's nested inside of my project, so I wonder if this could be an issue?

This is my setup.py file:

from setuptools import setup

setup(
    name='dataaggpackage',
    version='test',
    packages=['data_aggregation'],
    url='',
    license='test',
    author='czept',
    author_email='test',
    description='test'
)

However, trying to build a package(py -m build) using it, results in:

So then, I change my setup file to this, assuming that it would pick up the data_agreggation dir as a root.

from setuptools import setup

setup(
    name='dataaggpackage',
    version='test',
    packages=[''],
    url='',
    license='test',
    author='czept',
    author_email='test',
    description='test'
)

It builds correctly, I can install it on databricks cluster, but cannot import it.

user_3135332 · Answer

I believe it is because your setup.py file should be at the root of your project. Since it is in the data_aggregation folder, it can't find it. On your second setup, it builds but you don't have any packages in your setup.py

I would suggest using setuptools.find_packages(). It will go through your directory and look for init.py files. Your project should be organized like this :

project_root_directory
├── pyproject.toml  # AND/OR setup.cfg, setup.py
├── ...
└── dataaggpackage/
    ├── __init__.py
    ├── ...
    ├── module.py
    ├── subpkg1/
    │   ├── __init__.py
    │   ├── ...
    │   └── module1.py
    └── subpkg2/
        ├── __init__.py
        ├── ...
        └── module2.py

and your setup.py :

import setuptools
 
setup(
    name='dataaggpackage',
    version='test',
    packages=setuptools.find_packages(),
    url='',
    license='test',
    author='czept',
    author_email='test',
    description='test'
)

Check out setuptools official documentation for reference.

Databricks: cannot import custom python module

1 Answers