Creating a dynamic partial callable object in Python

user_5747500 0
Viewed 444

I have a regular expression defined in a YAML configuration file.

To make things easier, I'll use a dictionary here instead:

rule_1 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        'string': 's_test.log',
    }
}

I want to be able to parse that rule in a function.

If we assume such values don't change, then I could do something like this.

Importing modules:

import re
from operator import methodcaller
from functools import partial

My first function below is able to adapt to changes in the regex method used:

def rule_parser_re_1(*, kind, method, args=None, kwargs=None):
    if args is None: args = []
    if kwargs is None: kwargs = {}
    mc = methodcaller(method, **kwargs)
    return mc(re)

It works as expected:

>>> rule_parser_re_1(**rule_1)
<re.Match object; span=(0, 6), match='s_test'>

Now, let's say I don't have the string to parse available at the time the configuration dictionary is defined.

e.g. Let's say it's a specific line in a file which is accessible at runtime only.

myfile = """
first line
second line
third line
"""

io_myfile = io.StringIO(myfile)

content = io_myfile.readlines()

My second rule, where "line_number" (i.e. an int) replaces "string" (i.e. a str).

rule_2 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        'line_number': 2,
    }
}

My understanding is that I should be able to solve this by defining a partial rule_parser_re function. Such function should behave like the original one called with pattern and flags, but without string.

I've come up with the below function:

def rule_parser_re_2(*, kind, method, args=None, kwargs=None):
    if args is None: args = []
    if kwargs is None: kwargs = {}

    if kind == 'regex' and method == 'match':
        pa = partial(re.match, pattern=kwargs['pattern'], flags=kwargs['flags'])
        return pa

Which also seems to work properly:

>>> r2 = rule_parser_re_2(**rule_2)
>>> r2(string=content[2])
<re.Match object; span=(0, 6), match='second'>

Although, I see two maintainability problems with the above implementation:

  1. I'm using that if statement which forces me to amend the function for every re method I want to support;
  2. I need to explicitly specify the arguments, instead of just unpacking "**kwargs"

My aims/doubts:

  • Is there any way to make the above function more dynamic and maintainable?
  • Are functools.partial() and operator.methodcaller() the right tools for the job?
  • If so, can they be combined together?

Thanks!

functools python
3 Answers

Rather than trying a partial or methodcaller, why not call the function directly, using only kwargs, but use the configuration to drive most of the kwargs/args contents? I use a closure for that, where the prepped "remembers" the configuration.

Notice that my final call does not care that string is the keyword for re.match. I found that your example has a fair bit of coupling to regex specific stuff, some of which like re.X could not be stored in a YAML without further manipulation.

Likewise, the partial/methodcaller way to call the function should not have to care which line number in a file the value comes from, that is too much coupling. If you must, add something else in the config, not under kwargs, that deals with runtime parameter acquisition.

So I changed things around a bit. I believe, but you may disagree, that when calling a parse rule, the calling function should not have to know how the argument is called. Well, that is, unless you rules are only regex in style, in which case you don't need a kind in the config.

This is a quick, imperfect, sketch of an alternative approach. Details will depend on how exactly you want to use this.

I also punted on the *args handling, though it could probably carried out the same way if you had to.

import importlib

rule_1 = {
    'kind': 're',
    'method': 'match',
    'args': None,
    "positional_mapper" : ["string"],
    'kwargs': {
        'pattern': "[a-z_]+",
        # I don't know how this would be stored in a YAML
        # 'flags': re.X,
        'string': 's_test.log',
    }
}

rule_2 = {
    'kind': 're',
    'method': 'match',
    'args': None,
    "positional_mapper" : ["string"],
    'kwargs': {
        'pattern': "[a-z_]+",
    }
}


def prep(config):

    mod = app_urls = importlib.import_module(config["kind"])
    f = getattr(mod, config["method"])

    pre_args = config.get("args") or []
    pre_kwargs = config.get("kwargs") or {}
    positional_mapper = config["positional_mapper"]

    def prepped(*args, **kwargs):

        kwargs2 = pre_kwargs.copy()

        for value, argname in zip(args, positional_mapper):
            kwargs2[argname] = value
        kwargs2.update(**kwargs)

        return f(**kwargs2)

    return prepped


parsed_rule1 = prep(rule_1)

print ("#1", parsed_rule1("second line"))
print ("#2", parsed_rule1())

parsed_rule2 = prep(rule_2)
print ("#3", parsed_rule2("second line"))
print ("#3.5", parsed_rule2(string="second line"))
print ("#4", parsed_rule2())

As expected, call #4 chokes as it is missing an argument to put into string.

#1 <re.Match object; span=(0, 6), match='second'>
#2 <re.Match object; span=(0, 6), match='s_test'>
#3 <re.Match object; span=(0, 6), match='second'>
#3.5 <re.Match object; span=(0, 6), match='second'>
Traceback (most recent call last):
  File "test_299_dyn.py:57", in <module>
    print ("#4", parsed_rule2())
  File "test_299_dyn.py:44", in prepped
    return f(**kwargs2)
TypeError: match() missing 1 required positional argument: 'string'
user_13943530

You don't have to create a partial function. You can compile the pattern first then call the desired method with that:

rule_2 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        # 'line_number': 2, commented out this line
    }
}

content = ['', 'first line', 'second line', 'third line']

pattern = re.compile(**rule_2['kwargs'])
method = getattr(pattern, rule_2['method'])
>>> method(content[2])
<re.Match object; span=(0, 6), match='second'>

If you want to keep the line number, you can do something like this:

rule_2 = {
    'kind': 'regex',
    'method': 'match',
    'args': None,
    'kwargs': {
        'pattern': "[a-z_]+",
        'flags': re.X,
        'line_number': 2,
    }
}

content = ['', 'first line', 'second line', 'third line']
def rule_parser_re(*, kind, method, args=None, kwargs=None):
    copied_kwargs = kwargs.copy()
    line_number = copied_kwargs.pop('line_number')
    pattern = re.compile(**copied_kwargs)
    method = getattr(pattern, method)
    return method, line_number
    
parser, line_number = rule_parser_re(**rule_2)
>>> parser(content[line_number])
<re.Match object; span=(0, 6), match='second'>
user_96087590

Since your second schema doesn’t match the signature of re.match (etc.), you need to write your own function. It can use a wrapper function with named arguments to adapt the interface (although this involves fixing a position for your invented line_number argument if you care about args). It can also use getattr, which is equivalent to certain trivial uses of operator.methodcaller:

def rule2(kind,method,args,kwargs):
  return _rule2(getattr(re,method),*args or (),**kwargs or {})
def _rule2(f,pattern,line_number,flags):
  return lambda content: f(pattern,content[line_number],flags)

Note that content is the parameter that remains, since having only the line number leaves the file contents unknown; since it is not directly a parameter for the underlying function, partial isn’t the right tool here.

user_85862270
Related