Recursive globbing of fast-changing directories

user_11121626 0
Viewed 111

I stumbled upon unintuitive problem: if you recursively glob (with pathlib) fast-changing directory, where other directories are created and deleted, you have a chance to get FileNotFoundException. If directory is not found, then why bother me with this? Why not just skip bogus directory and continue? And how i can get around this in a compact way?

Here is how to reproduce this:

Launch one terminal with this sh command

while true; do mkdir deleteme && rmdir deleteme; done

and another with this python script:

from pathlib import Path
p = Path()
while True:
    list(p.rglob('*'))

or, as oneliner:

while true; do mkdir 1 && rmdir 1; done & python3 -c "from pathlib import Path; from itertools import count; p = Path(); [list(p.rglob('*')) for i in count(start=1)]"; fg
pathlib glob python-3.x python
1 Answers

The primitive generating the exception is os.scandir.

What is happening is the pathlib.Path.rglob() method makes a selector object (pathlib._WildcardSelector via pathlib._make_selector, given your pattern), and then this is lazily iterated over during the execution of the rglob() generator.

The pathlib module doesn't seem to give any hooks to readily customize or override this behavior.

The best solution would probably be to subclass the Path or PosixPath class, override the rglob() method, rebuilding the implementation in a way that is resilient to this error mode.

A monkey patch method that seems to work as a proof of concept:

import contextlib
import pathlib

def my_scandir(path=None):
     while True:
         try:
             result = list(os.scandir(path))
         except FileNotFoundError:
             continue
         @contextlib.contextmanager
         def wrapped_result():
             try:
                 yield result
             finally:
                 pass
         return wrapped_result()

pathlib._NormalAccessor.scandir = staticmethod(my_scandir)

After this patch is applied, the while loop with rglob('*') in your example no longer fails with FileNotFoundError.

This works on Python v3.8.5. Because it uses implementation details of the pathlib module, it may not work on other python versions.

user_1969700
Related