Use of subprocess with Linux pipe command

user_102221 0
Viewed 228

I want to run from Python script next command:

strings <FILE NAME> | grep "Version = <VERSION STRING>" > /dev/null

I need to save command return code and command output for following script logic.

Currently I used next code:

     strings_out = subprocess.Popen(('strings', file), stdout=subprocess.PIPE)
     grep_output = subprocess.check_output(('grep', "Version = " + version_string), stdin=strings_out.stdout)
     strings_out.wait()

I get error

subprocess.CalledProcessError: Command '('grep', 'Version = <VERSION STRING>')' returned non-zero exit status 1

My assumption is that check_output run out of memory.

What is wrong in my use of subprocess?

subprocess python
3 Answers

A non-zero exit status for a check_output means that the bash command had a problem - I don't think you ran out of memory.

On testing myself, I found that if I gave grep a string that exists within a file, I got a proper output with your code (I'm not using Version because I don't know what input files you have, but otherwise things are just about the same). I do, however, get the same error you get if I grep a string that doesn't exist.

Maybe you are running it on a file that string doesn't output any instances of "Version = " + version_string. If you are in a loop, it would only take one file to not have the proper string to get the error.

On another note, if you plan on finishing this line: strings <FILE NAME> | grep "Version = <VERSION STRING>" > /dev/null with subprocess, you'll be piping the output to /dev/null. In this case, you won't see the output of grep.

user_140922600

As @samsonjm has mentioned, every successfully ran bash command has the exit code = 0. It implies that the grep command has failed. Moreover, there is no clue for an OutofMemory error.

I suspect that input file to the strings command is large and hence it could be taking more time to return its result. Therefore, I suspect the string_out.wait() directive should be called immediately after the first line above to feed in the input from stdin to the grep command. It is reasonable to think in this way as the subprocess executes commands in a child process that might be running until completion.

strings_out = subprocess.Popen(('strings', file), stdout=subprocess.PIPE) 
strings_out.wait()
grep_output = subprocess.check_output(('grep', "Version = " + version_string), stdin=strings_out.stdout)
user_53978450

That's neat, I've never thought to use subprocess stdin/stdout like that before. However, my advice would be to either go pure Python and write a method to search for the string in a file, or get a little fancier with your subprocess line.

Python might look something like:

import os
search_term = bytes("Version = " + version_string, encoding='utf-8')
i = 0
found = False
file_size = os.stat(f).st_size
chunk_size = len(search_term) *10
with open(file_name, 'rb') as f:
    while f.tell() < size:
        x = f.read() #read a small amount of data
        i += chunk_size - len(search_term) #to make sure we don't miss the search_term
        f.seek(i)
        if search_term in x:
            found = True
            break

For subprocess:

cmd = f'strings {file_name} | grep "Version = {version_string}"'
test = subprocess.run([cmd], shell=True, capture_output=True)
test.returncode
user_125179500
Related