Numba RuntimeError only when directly running on databricks notebook

user_186078 0
Viewed 358

I am trying to understand the source of runtime error when I run the following python function in a databricks notebook vs importing and invoking it from a module

Running directly on databricks notebook

def test_numba_func():
    from numba import jit

    @jit(cache=True)  
    def test():
        return .5 ** 2 / 4.0

    print(test())

invoking this function does not work

test_numba_func()

RuntimeError: cannot cache function 'test_numba_func..test': no locator available for file ''

However if I create a module say databricks_test.py with the same function, then the following import works without any issues.

Module import

import databricks_test
databricks_test.test_numba_func()

Databricks notebook Databricks notebook screenshot

I am able to run in directly in Colab though.

enter image description here

I think this is due to some rights issue. How can I fix my code to make it work against databricks?

azure-databricks databricks
1 Answers

The difference is that notebooks on Databricks aren't the real files - they are kept in memory when you're using them, and persisted to something like Database on the fly. This is different from the using the module that is real file, or Google notebooks where they are also files on the disk.

Theoretically you can try to set environment variable NUMBA_CACHE_DIR to something like /tmp/cache (docs), but I'm not sure that it will work. Also, it will work only until cluster is terminated.

Alex_Ott_186270
Related