mkstemp opening too many files

user_6225525 0
Viewed 856

I'm using subprocess.run in a loop (more than 10 000 times) to call some java command. Like this:

 import subprocess
 import tempfile

 for i in range(10000):
     ret = subprocess.run(["ls"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 
     (_, name) = tempfile.mkstemp()
     with open(name, 'w+') as fp:
         fp.write(ret.stdout.decode())

However, after some time, I got the following exception:

Traceback (most recent call last):
  File "mwe.py", line 5, in <module>
    ret = subprocess.run(["ls"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 
  File "/usr/lib/python3.5/subprocess.py", line 693, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1454, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

Am I missing something to close some file descriptor? Thanks

subprocess python
1 Answers

mkstemp returns an already open file descriptor fd followed by the filename. You are ignoring the file descriptor (your choice of the name _ suggests you have explicitly chosen to ignore it) and as a result you are neglecting to close it. Instead, you open the file a second time using the filename, creating a file object that contains a second file descriptor for the same file. Regardless of whether you close that second one, the first one remains open.

Here's a fix to the mkstemp approach:

 temporaryFiles = []
 for i in range(1000):
     ...
     fd, name = tempfile.mkstemp()
     os.write(fd, ... )
     os.close(fd)
     temporaryFiles.append(name)  # remember the filename for future processing/deletion

Building on Wyrmwood's suggestion in the comments, an even better approach would be:

 temporaryFiles = []
 for i in range(1000):
     ...
     with tempfile.NamedTemporaryFile(delete=False) as tmp:
         # tmp is a context manager that will automatically close the file when you exit this clause
         tmp.file.write( ... )
         temporaryFiles.append(tmp.name)  # remember the filename for future processing/deletion

Note that both mkstemp and the NamedTemporaryFile constructor have arguments that allow you to be more specific about the file's location (dir) and naming (prefix, suffix). If you want to keep the files, you should specify dir so that you keep them out of the default temporary directory, since the default location may get cleaned up by the OS.

user_30196890
Related