catching stdout in realtime from subprocess

user_194532 0
Viewed 151734

I want to subprocess.Popen() rsync.exe in Windows, and print the stdout in Python.

My code works, but it doesn't catch the progress until a file transfer is done! I want to print the progress for each file in real time.

Using Python 3.1 now since I heard it should be better at handling IO.

import subprocess, time, os, sys

cmd = "rsync.exe -vaz -P source/ dest/"
p, line = True, 'start'


p = subprocess.Popen(cmd,
                     shell=True,
                     bufsize=64,
                     stdin=subprocess.PIPE,
                     stderr=subprocess.PIPE,
                     stdout=subprocess.PIPE)

for line in p.stdout:
    print(">>> " + str(line.rstrip()))
    p.stdout.flush()
subprocess stdout python
13 Answers

Depending on the use case, you might also want to disable the buffering in the subprocess itself.

If the subprocess will be a Python process, you could do this before the call:

os.environ["PYTHONUNBUFFERED"] = "1"

Or alternatively pass this in the env argument to Popen.

Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:

cmd = ["stdbuf", "-oL"] + cmd

See also here about stdbuf or other options.

user_1333740

if you run something like this in a thread and save the ffmpeg_time property in a property of a method so you can access it, it would work very nice I get outputs like this: output be like if you use threading in tkinter

input = 'path/input_file.mp4'
output = 'path/input_file.mp4'
command = "ffmpeg -y -v quiet -stats -i \"" + str(input) + "\" -metadata title=\"@alaa_sanatisharif\" -preset ultrafast -vcodec copy -r 50 -vsync 1 -async 1 \"" + output + "\""
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, shell=True)
for line in self.process.stdout:
    reg = re.search('\d\d:\d\d:\d\d', line)
    ffmpeg_time = reg.group(0) if reg else ''
    print(ffmpeg_time)
user_131761490

In Python 3, here's a solution, which takes a command off the command line and delivers real-time nicely decoded strings as they are received.

Receiver (receiver.py):

import subprocess
import sys

cmd = sys.argv[1:]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in p.stdout:
    print("received: {}".format(line.rstrip().decode("utf-8")))

Example simple program that could generate real-time output (dummy_out.py):

import time
import sys

for i in range(5):
    print("hello {}".format(i))
    sys.stdout.flush()  
    time.sleep(1)

Output:

$python receiver.py python dummy_out.py
received: hello 0
received: hello 1
received: hello 2
received: hello 3
received: hello 4
user_6958040
Related