Lazy module variables--can it be done?

user_18763 0
Viewed 15629

I'm trying to find a way to lazily load a module-level variable.

Specifically, I've written a tiny Python library to talk to iTunes, and I want to have a DOWNLOAD_FOLDER_PATH module variable. Unfortunately, iTunes won't tell you where its download folder is, so I've written a function that grabs the filepath of a few podcast tracks and climbs back up the directory tree until it finds the "Downloads" directory.

This takes a second or two, so I'd like to have it evaluated lazily, rather than at module import time.

Is there any way to lazily assign a module variable when it's first accessed or will I have to rely on a function?

itunes lazy-loading module python variables
8 Answers

It turns out that as of Python 3.7, it's possible to do this cleanly by defining a __getattr__() at the module level, as specified in PEP 562 and documented in the data model chapter in the Python reference documentation.

# mymodule.py

from typing import Any

DOWNLOAD_FOLDER_PATH: str

def _download_folder_path() -> str:
    global DOWNLOAD_FOLDER_PATH
    DOWNLOAD_FOLDER_PATH = ... # compute however ...
    return DOWNLOAD_FOLDER_PATH

def __getattr__(name: str) -> Any:
    if name == "DOWNLOAD_FOLDER_PATH":
        return _download_folder_path()
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
user_10090650

The proper way of doing this, according to the Python docs, is to subclass types.ModuleType and then dynamically update the module's __class__. So, here's a solution loosely on Christian Tismer's answer but probably not resembling it much at all:

import sys
import types

class _Sneaky(types.ModuleType):
    @property
    def DOWNLOAD_FOLDER_PATH(self):
        if not hasattr(self, '_download_folder_path'):
            self._download_folder_path = '/dev/block/'
        return self._download_folder_path
sys.modules[__name__].__class__ = _Sneaky
user_52237570

Since Python 3.7 (and as a result of PEP-562), this is now possible with the module-level __getattr__:

Inside your module, put something like:

def _long_function():
    # print() function to show this is called only once
    print("Determining DOWNLOAD_FOLDER_PATH...")
    # Determine the module-level variable
    path = "/some/path/here"
    # Set the global (module scope)
    globals()['DOWNLOAD_FOLDER_PATH'] = path
    # ... and return it
    return path


def __getattr__(name):
    if name == "DOWNLOAD_FOLDER_PATH":
        return _long_function()

    # Implicit else
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

From this it should be clear that the _long_function() isn't executed when you import your module, e.g.:

print("-- before import --")
import somemodule
print("-- after import --")

results in just:

-- before import --
-- after import --

But when you attempt to access the name from the module, the module-level __getattr__ will be called, which in turn will call _long_function, which will perform the long-running task, cache it as a module-level variable, and return the result back to the code that called it.

For example, with the first block above inside the module "somemodule.py", the following code:

import somemodule
print("--")
print(somemodule.DOWNLOAD_FOLDER_PATH)
print('--')
print(somemodule.DOWNLOAD_FOLDER_PATH)
print('--')

produces:

--
Determining DOWNLOAD_FOLDER_PATH...
/some/path/here
--
/some/path/here
--

or, more clearly:

# LINE OF CODE                                # OUTPUT
import somemodule                             # (nothing)

print("--")                                   # --

print(somemodule.DOWNLOAD_FOLDER_PATH)        # Determining DOWNLOAD_FOLDER_PATH...
                                              # /some/path/here

print("--")                                   # --

print(somemodule.DOWNLOAD_FOLDER_PATH)        # /some/path/here

print("--")                                   # --

Lastly, you can also implement __dir__ as the PEP describes if you want to indicate (e.g. to code introspection tools) that DOWNLOAD_FOLDER_PATH is available.

user_7369370
Related