Building a minimal plugin architecture in Python

Viewed 105583

I have an application, written in Python, which is used by a fairly technical audience (scientists).

I'm looking for a good way to make the application extensible by the users, i.e. a scripting/plugin architecture.

I am looking for something extremely lightweight. Most scripts, or plugins, are not going to be developed and distributed by a third-party and installed, but are going to be something whipped up by a user in a few minutes to automate a repeating task, add support for a file format, etc. So plugins should have the absolute minimum boilerplate code, and require no 'installation' other than copying to a folder (so something like setuptools entry points, or the Zope plugin architecture seems like too much.)

Are there any systems like this already out there, or any projects that implement a similar scheme that I should look at for ideas / inspiration?

18 Answers

Mine is, basically, a directory called "plugins" which the main app can poll and then use imp.load_module to pick up files, look for a well-known entry point possibly with module-level config params, and go from there. I use file-monitoring stuff for a certain amount of dynamism in which plugins are active, but that's a nice-to-have.

Of course, any requirement that comes along saying "I don't need [big, complicated thing] X; I just want something lightweight" runs the risk of re-implementing X one discovered requirement at a time. But that's not to say you can't have some fun doing it anyway :)

module_example.py:

def plugin_main(*args, **kwargs):
    print args, kwargs

loader.py:

def load_plugin(name):
    mod = __import__("module_%s" % name)
    return mod

def call_plugin(name, *args, **kwargs):
    plugin = load_plugin(name)
    plugin.plugin_main(*args, **kwargs)

call_plugin("example", 1234)

It's certainly "minimal", it has absolutely no error checking, probably countless security problems, it's not very flexible - but it should show you how simple a plugin system in Python can be..

You probably want to look into the imp module too, although you can do a lot with just __import__, os.listdir and some string manipulation.

While that question is really interesting, I think it's fairly hard to answer, without more details. What sort of application is this? Does it have a GUI? Is it a command-line tool? A set of scripts? A program with an unique entry point, etc...

Given the little information I have, I will answer in a very generic manner.

What means do you have to add plugins?

  • You will probably have to add a configuration file, which will list the paths/directories to load.
  • Another way would be to say "any files in that plugin/ directory will be loaded", but it has the inconvenient to require your users to move around files.
  • A last, intermediate option would be to require all plugins to be in the same plugin/ folder, and then to active/deactivate them using relative paths in a config file.

On a pure code/design practice, you'll have to determine clearly what behavior/specific actions you want your users to extend. Identify the common entry point/a set of functionalities that will always be overridden, and determine groups within these actions. Once this is done, it should be easy to extend your application,

Example using hooks, inspired from MediaWiki (PHP, but does language really matters?):

import hooks

# In your core code, on key points, you allow user to run actions:
def compute(...):
    try:
        hooks.runHook(hooks.registered.beforeCompute)
    except hooks.hookException:
        print('Error while executing plugin')

    # [compute main code] ...

    try:
        hooks.runHook(hooks.registered.afterCompute)
    except hooks.hookException:
        print('Error while executing plugin')

# The idea is to insert possibilities for users to extend the behavior 
# where it matters.
# If you need to, pass context parameters to runHook. Remember that
# runHook can be defined as a runHook(*args, **kwargs) function, not
# requiring you to define a common interface for *all* hooks. Quite flexible :)

# --------------------

# And in the plugin code:
# [...] plugin magic
def doStuff():
    # ....
# and register the functionalities in hooks

# doStuff will be called at the end of each core.compute() call
hooks.registered.afterCompute.append(doStuff)

Another example, inspired from mercurial. Here, extensions only add commands to the hg commandline executable, extending the behavior.

def doStuff(ui, repo, *args, **kwargs):
    # when called, a extension function always receives:
    # * an ui object (user interface, prints, warnings, etc)
    # * a repository object (main object from which most operations are doable)
    # * command-line arguments that were not used by the core program

    doMoreMagicStuff()
    obj = maybeCreateSomeObjects()

# each extension defines a commands dictionary in the main extension file
commands = { 'newcommand': doStuff }

For both approaches, you might need common initialize and finalize for your extension. You can either use a common interface that all your extension will have to implement (fits better with second approach; mercurial uses a reposetup(ui, repo) that is called for all extension), or use a hook-kind of approach, with a hooks.setup hook.

But again, if you want more useful answers, you'll have to narrow down your question ;)

I am a retired biologist who dealt with digital micrograqphs and found himself having to write an image processing and analysis package (not technically a library) to run on an SGi machine. I wrote the code in C and used Tcl for the scripting language. The GUI, such as it was, was done using Tk. The commands that appeared in Tcl were of the form "extensionName commandName arg0 arg1 ... param0 param1 ...", that is, simple space-separated words and numbers. When Tcl saw the "extensionName" substring, control was passed to the C package. That in turn ran the command through a lexer/parser (done in lex/yacc) and then called C routines as necessary.

The commands to operate the package could be run one by one via a window in the GUI, but batch jobs were done by editing text files which were valid Tcl scripts; you'd pick the template that did the kind of file-level operation you wanted to do and then edit a copy to contain the actual directory and file names plus the package commands. It worked like a charm. Until ...

1) The world turned to PCs and 2) the scripts got longer than about 500 lines, when Tcl's iffy organizational capabilities started to become a real inconvenience. Time passed ...

I retired, Python got invented, and it looked like the perfect successor to Tcl. Now, I have never done the port, because I have never faced up to the challenges of compiling (pretty big) C programs on a PC, extending Python with a C package, and doing GUIs in Python/Gt?/Tk?/??. However, the old idea of having editable template scripts seems still workable. Also, it should not be too great a burden to enter package commands in a native Python form, e.g.:

packageName.command( arg0, arg1, ..., param0, param1, ...)

A few extra dots, parens, and commas, but those aren't showstoppers.

I remember seeing that someone has done versions of lex and yacc in Python (try: http://www.dabeaz.com/ply/), so if those are still needed, they're around.

The point of this rambling is that it has seemed to me that Python itself IS the desired "lightweight" front end usable by scientists. I'm curious to know why you think that it is not, and I mean that seriously.


added later: The application gedit anticipates plugins being added and their site has about the clearest explanation of a simple plugin procedure I've found in a few minutes of looking around. Try:

https://wiki.gnome.org/Apps/Gedit/PythonPluginHowToOld

I'd still like to understand your question better. I am unclear whether you 1) want scientists to be able to use your (Python) application quite simply in various ways or 2) want to allow the scientists to add new capabilities to your application. Choice #1 is the situation we faced with the images and that led us to use generic scripts which we modified to suit the need of the moment. Is it Choice #2 which leads you to the idea of plugins, or is it some aspect of your application that makes issuing commands to it impracticable?

I enjoyed the nice discussion on different plugin architectures given by Dr Andre Roberge at Pycon 2009. He gives a good overview of different ways of implementing plugins, starting from something really simple.

Its available as a podcast (second part following an explanation of monkey-patching) accompanied by a series of six blog entries.

I recommend giving it a quick listen before you make a decision.

You can use pluginlib.

Plugins are easy to create and can be loaded from other packages, file paths, or entry points.

Create a plugin parent class, defining any required methods:

import pluginlib

@pluginlib.Parent('parser')
class Parser(object):

    @pluginlib.abstractmethod
    def parse(self, string):
        pass

Create a plugin by inheriting a parent class:

import json

class JSON(Parser):
    _alias_ = 'json'

    def parse(self, string):
        return json.loads(string)

Load the plugins:

loader = pluginlib.PluginLoader(modules=['sample_plugins'])
plugins = loader.plugins
parser = plugins.parser.json()
print(parser.parse('{"json": "test"}'))

You may also have a look at Groundwork.

The idea is to build applications around reusable components, called patterns and plugins. Plugins are classes that derive from GwBasePattern. Here's a basic example:

from groundwork import App
from groundwork.patterns import GwBasePattern

class MyPlugin(GwBasePattern):
    def __init__(self, app, **kwargs):
        self.name = "My Plugin"
        super().__init__(app, **kwargs)

    def activate(self): 
        pass

    def deactivate(self):
        pass

my_app = App(plugins=[MyPlugin])       # register plugin
my_app.plugins.activate(["My Plugin"]) # activate it

There are also more advanced patterns to handle e.g. command line interfaces, signaling or shared objects.

Groundwork finds its plugins either by programmatically binding them to an app as shown above or automatically via setuptools. Python packages containing plugins must declare these using a special entry point groundwork.plugin.

Here are the docs.

Disclaimer: I'm one of the authors of Groundwork.

In our current healthcare product we have a plugin architecture implemented with interface class. Our tech stack are Django on top of Python for API and Nuxtjs on top of nodejs for frontend.

We have a plugin manager app written for our product which is basically pip and npm package in adherence with Django and Nuxtjs.

For new plugin development(pip and npm) we made plugin manager as dependency.

In Pip package: With the help of setup.py you can add entrypoint of the plugin to do something with plugin manager(registry, initiations, ...etc.) https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

In npm package: Similar to pip there are hooks in npm scripts to handle the installation. https://docs.npmjs.com/misc/scripts

Our usecase:

plugin development team is separate from core devopment team now. The scope of plugin development is for integrating with 3rd party apps which are defined in any of the categories of the product. The plugin interfaces are categorised for eg:- Fax, phone, email ...etc plugin manager can be enhanced to new categories.

In your case: Maybe you can have one plugin written and reuse the same for doing stuffs.

If plugin developers has need to use reuse core objects that object can be used by doing a level of abstraction within plugin manager so that any plugins can inherit those methods.

Just sharing how we implemented in our product hope it will give a little idea.

Related