How to read python files from a directory and search for functions

user_8850098 0
Viewed 52

I am looking to make a dictionary of python file names as the value and the functions defined in each file as the key. I am easily able to make a list of the filenames. For example:

    for child in Path('./loaders').iterdir():
        if child.is_file():
            f = child.name
            if f != "__init__.py":
                self.beamline_options.append(f.split('.')[0])

I also have a variable defined as self.beamline_loaders = {}, which I would like to add to this loop above in such a way:

    for child in Path('./loaders').iterdir():
        if child.is_file():
            f = child.name
            if f != "__init__.py":
                self.beamline_options.append(f.split('.')[0])
                self.beamline_loaders[f] = [def1, def2, def3, ...]

where def1, def2, ... would be the names of the functions in each file. All of this is within the same module. I am using this dictionary to populate a QComboBox based on the selection of a first combo-box (which consists of just the names in the list of beamline options). This could in theory be hard coded, but I would prefer not to hard code it.

qcombobox pyqt search function python
2 Answers

It sounds like you're asking for a simple plugin system that can dynamically load functions from any python files found in a target directory. Presumably, you will also need to execute those functions at some point, rather than just display their names. In which case, one solution would be to use importlib to directly import the modules from the source files, and then use dir to scan the module's attributes for suitable function objects.

Below is a basic demo that implements that, and also shows how to populate some combo-boxes with the module and function names. The function objects themselves are loaded on demand and stored in a dict, and can be displayed via the "Load" button. (NB: the demo assumes that the "loaders" directory is in the same directory as the demo script):

screenshot

import sys, types, importlib.util
from collections import defaultdict
from pathlib import Path
from PyQt5 import QtWidgets

BASEDIR = Path(__file__).resolve().parent.joinpath('loaders')

class Window(QtWidgets.QWidget):
    def __init__(self):
        super().__init__()
        self.buttonLoad = QtWidgets.QPushButton('Load')
        self.buttonLoad.clicked.connect(self.handleLoad)
        self.comboOptions = QtWidgets.QComboBox()
        self.comboOptions.currentTextChanged.connect(self.handleOptions)
        self.comboLoaders = QtWidgets.QComboBox()
        layout = QtWidgets.QHBoxLayout(self)
        layout.addWidget(self.comboOptions)
        layout.addWidget(self.comboLoaders)
        layout.addWidget(self.buttonLoad)
        self.scanOptions()

    def scanOptions(self):
        self.beamline_options = {}
        self.beamline_loaders = defaultdict(dict)
        for entry in BASEDIR.iterdir():
            if entry.is_file() and entry.match('[!_.]*.py'):
                self.beamline_options[entry.stem] = entry
        self.comboOptions.addItems(self.beamline_options)

    def getLoaders(self, option):
        if option not in self.beamline_loaders:
            entry = self.beamline_options[option]
            modname = f'{BASEDIR.name}.{option}'
            spec = importlib.util.spec_from_file_location(modname, entry)
            module = importlib.util.module_from_spec(spec)
            sys.modules[modname] = module
            spec.loader.exec_module(module)
            for name in dir(module):
                if not name.startswith('_'):
                    obj = getattr(module, name)
                    if isinstance(obj, types.FunctionType):
                        self.beamline_loaders[option][name] = obj
        return self.beamline_loaders[option]

    def getLoader(self, option, loader):
        if option and loader:
            return self.getLoaders(option)[loader]

    def handleOptions(self, option):
        self.comboLoaders.clear()
        self.comboLoaders.addItems(self.getLoaders(option))

    def handleLoad(self):
        option = self.comboOptions.currentText()
        loader = self.comboLoaders.currentText()
        func = self.getLoader(option, loader)
        QtWidgets.QMessageBox.information(self, 'Load', repr(func))

if __name__ == '__main__':

    app = QtWidgets.QApplication(['Test'])
    window = Window()
    window.setGeometry(600, 100, 400, 50)
    window.show()
    app.exec_()
user_9844210

One possible means of doing this would be to use the ast module built in to python.

With the ast module you could read a python file and parse it into individual python language tokens, and collect the ones that identify as function definitions.

for example:

import ast

def parse_file(path):
    source = path.read_text()
    tree = ast.parse(source)
    top_level = []
    for stmt in ast.iter_child_nodes(tree):
        if isinstance(stmt, ast.FunctionDef) or isinstance(stmt, ast.ClassDef):
            top_level.append(stmt.name)
    return top_level
    
for child in Path('./loaders').iterdir():
    if child.is_file() and child.suffix == ".py":
        if child.name != "__init__.py":
            self.beamline_options.append(child.name.split('.')[0])
            self.beamline_loaders[child.name] = parse_file(child)
user_178294510
Related