check what files are open in Python

Viewed 56710

I'm getting an error in a program that is supposed to run for a long time that too many files are open. Is there any way I can keep track of which files are open so I can print that list out occasionally and see where the problem is?

11 Answers

I ended up wrapping the built-in file object at the entry point of my program. I found out that I wasn't closing my loggers.

import io
import sys
import builtins
import traceback
from functools import wraps


def opener(old_open):
    @wraps(old_open)
    def tracking_open(*args, **kw):
        file = old_open(*args, **kw)

        old_close = file.close
        @wraps(old_close)
        def close():
            old_close()
            open_files.remove(file)
        file.close = close
        file.stack = traceback.extract_stack()

        open_files.add(file)
        return file
    return tracking_open


def print_open_files():
    print(f'### {len(open_files)} OPEN FILES: [{", ".join(f.name for f in open_files)}]', file=sys.stderr)
    for file in open_files:
        print(f'Open file {file.name}:\n{"".join(traceback.format_list(file.stack))}', file=sys.stderr)


open_files = set()
io.open = opener(io.open)
builtins.open = opener(builtins.open)

On Linux, you can look at the contents of /proc/self/fd:

$ ls -l /proc/self/fd/
total 0
lrwx------ 1 foo users 64 Jan  7 15:15 0 -> /dev/pts/3
lrwx------ 1 foo users 64 Jan  7 15:15 1 -> /dev/pts/3
lrwx------ 1 foo users 64 Jan  7 15:15 2 -> /dev/pts/3
lr-x------ 1 foo users 64 Jan  7 15:15 3 -> /proc/9527/fd

On Linux, you can use lsof to show all files opened by a process.

On Windows, you can use Process Explorer to show all file handles owned by a process.

I'd guess that you are leaking file descriptors. You probably want to look through your code to make sure that you are closing all of the files that you open.

Related