How to append the Output of a command to temporary file

Viewed 88

I have been trying to append my output of a command to a temporary file in python and later doing some operations. Not able to append the data to a temporary file. Any help is appreciated! My sample code as follows.

Getting the error like this.

with open(temp1 , 'r') as f: TypeError: expected str, bytes or os.PathLike object, not _TemporaryFileWrapper

import tempfile
import os
temp1 = tempfile.NamedTemporaryFile()
os.system("echo Hello world | tee temp1")
with open(temp1 , 'r') as f:
    a = f.readlines()[-1]
    print(a)
5 Answers
import tempfile
import os

# Opening in update-text mode to avoid encoding the data written to it
temp1 = tempfile.NamedTemporaryFile("w+")
# popen opens a pipe from the command, allowing one to capture its output
output = os.popen("echo Hello world")

# Write the command output to the temporary file
temp1.write(output.read())

# Reset the stream position at the beginning of the file, if you want to read its contents
temp1.seek(0)
print(temp1.read())

Check out subprocess.Popen for more powerful subprocess communication.

Whatever you're trying to do isn't right. It appears that you are trying to have a system call write to a file, and then you want to read that file in your Python code. You're creating a temporary file, but then your system call is writing to a statically named file, named 'temp1' rather than to the temporary file you've opened. So it's unclear if you want/need to use a computed temporary file name or if using temp1 is OK. The easiest way to fix your code to do what I think you want is like this:

import os
os.system("echo Hello world | tee temp1")
with open('temp1' , 'r') as f:
    a = f.readlines()[-1]
    print(a)

If you need to create a temporary file name in your situation, then you have to be careful if you are at all concerned about security or thread safety. What you really want to do is have the system create a temporary directory for you, and then create a statically named file in that directory. Here's your code reworked to do that:

import tempfile
import os

with tempfile.TemporaryDirectory() as dir:
    tempfile = os.path.join(dir, "temp1")
    os.system("echo Hello world /tmp > " + tempfile)
    with open(tempfile) as f:
        buf = f.read()

print(buf)

This method has the added benefit of automatically cleaning up for you.

UPDATE: I have now seen @UlisseBordingnon's answer. That's a better solution overall. Using os.system() is discouraged. I would have gone a bit different of a way by using the subprocess module, but what they suggest is 100% valid, and is thread and security safe. I guess I'll leave my answer here as maybe you or other readers need to use os.system() or otherwise have the shell process you execute write directly to a file.

As others have suggested, you should use the subprocess module instead of os.system. However from subprocess you can use the most recent interface (and by most recent, I believe this was adding in Python 3.4) of subprocess.run.

The neat thing about using .run is that you can pass any file-like object to stdout and the stdout stream will automatically redirect to that file.

import tempfile
import subprocess

with tempfile.NamedTemporaryFile("w+") as f:
    subprocess.run(["echo", "hello world"], stdout=f)
    
    # command has finished running, let's check the file
    f.seek(0)
    print(f.read())
    # hello world

If you are using python 3.5 or later (as with most of us), then use subprocess.run is better because you do not need a temporary file:

import subprocess

completed_process = subprocess.run(
    ["echo", "hello world"],
    capture_output=True,
    encoding="utf-8",
)
print(completed_process.stdout)

Notes

  • The capture_output parameter tells run() to save the output to the .stdout and .stderr attributes
  • The encoding parameter will convert the output from bytes to string

Depending on your needs, if your print your output, a quickier way, but maybe not exactly what you are looking for is to redirect the output to a file, at the command line level Example(egfile.py):

import os

os.system("echo Hello world")

At command level you can simply do:

python egfile.py > file.txt

The output of the file will be redirected to the file instead to the screen

Related