I have an interesting design problem and I wanted to ask the community for advice. I am designing a bunch of tools that deal with molecules, which are handled as a pickable class. The tools adhere to the KISS principle, so each is going to be specialized in one task only, and the idea is to allow to optionally combine them using Unix pipes to do something like this:
$> generator.py | filter.py [OPTS] | calc_properties.py
While each tool can be used independently, by allowing chaining multiple tools reduces redundancy and the amount of data to be saved and read at every step. Also, each tool can potentially generate more than enough data to saturate all the available RAM. I've looked around and while there is quite a lot of documentation about Python support for pipes, I can't find anything that tells me first if this is a good design pattern, and second what would be a good way to pass around pickable objects through the pipes.
EDIT: I think managed to put together a working solution and I'm sharing it here in case it can help someone else. It allows the flexibility of specifying input files or reading from stdin, and it is not complex, but it is not trivial either. For example the binary data cannot be written directly to sys.stdout (class _io.TextIOWrapper) but needs to be sent to sys.stdout.buffer (class _io.BufferedWriter).
--
# sender.py
import sys
import pickle
l = [1,2,3]
for x in l:
pickle.dump(x, sys.stdout.buffer)
--
# receiver.py
import sys
import pickle
interactive = not sys.stdin.isatty()
print("[ receiver is in a pipe: %s ] "% interactive)
if not interactive and len(sys.argv)<2:
print("ERROR: intput file must be specified")
sys.exit(1)
try:
while True:
x = pickle.load(sys.stdin.buffer)
print("RECEIVED", x)
except EOFError:
print("[EOF]")
The output:
$ python receive.py
[ receiver is in a pipe: False ]
ERROR: intput file must be specified
$ python send.py | python receive.py
[ receiver is in a pipe: True ]
RECEIVED 1
RECEIVED 2
RECEIVED 3
[EOF]