How to get insertions for unix-style wildcards in python?

user_13583564 0
Viewed 38

I have a somewhat complicated file-structure. Using glob.glob('base_directory/*/*/*<some_pattern>.dat') I do get a list of all the relevant filenames, but I would need to work with the names of the relevant subdirectories on the different levels, so what I really want are just the insertions for the wildcards.

In the end I would like to build a dictionary of dictionaries for the insertions at each sub-level of my file structure. But for a start I only need a function / package that returns only the insertions of the pattern-matching. Any ideas where to find that?

glob python
1 Answers

Assuming I'm reading the question correctly, you can get all the matching paths with pathlib.Path.glob:

pushd /tmp
mkdir -p a/b/c/d/1.dat
mkdir -p a/b/e/g/2.dat
mkdir -p a/b/c/d/e/g.dat

>>> from pathlib import Path
>>> files = list(Path("/tmp").glob("**/*.dat"))
>>> print(files)
[PosixPath('/tmp/a/b/e/g/2.dat'), PosixPath('/tmp/a/b/c/d/1.dat'), PosixPath('/tmp/a/b/c/d/e/g.dat')]

See the pathlib docs on glob for more details. There's also .rglob() which effectively saves typing **.

Extracting parts

Apparently you want something like groups (()) in regexs. Whilst there's no support for that in pathlib.glob of which I'm aware, for the particular example you gave it's easy enough to get it:

import re

def analyse_path(p: Path) -> tuple[str]:
    _*, a, b, c, fn = p.parts
    match_ = re.search("param1_([0-9]+)-param2_(.+).dat", fn)
    if not match_:
        raise ValueError(f"unable to extract data from {fn}") 
    return (a, b, c, *match_.groups())

useful_data = [analyse_path(p) for p in Path("/tmp").glob("*/*/*/*.dat")]

The idea here is to use the glob to narrow down and the do a regex search on the filename itself. But if you need to do anything more complicated than '*' in the glob, I would just do the whole thing with a regex, probably operating on the result of Path(basedir).glob("**"). If you have very large dirs or lots of irrelevant files you'll likely hit a bottlneck here, but profile before you try anything less clear than a big glob.

user_154526010
Related