Write() with multiprocessing: built-in locking to prevent garbled output?

Viewed 96

My apologies, everyone. Additional study convinces me that the parallel writing of a single file by multiple processes described below is much rarer than I originally thought, and so I don't feel nearly as confident that I have lots of experience with no ill effects. More likely I'm just getting lucky and the data pool is small enough that it just hasn't broken yet.

For posterity, this was my original question:

I'm hoping someone can help me understand what I'm seeing.

I started with a single process that writes multiple files. Over time it evolved into multiple processes (using ProcessPoolExecutor from concurrent.futures). During that evolution I unintentionally wound up with multiple processes writing to the same files in parallel.

What puzzles me is that it seems to be working fine. At least to date (and it's been months of generating millions of lines of data every day), I haven't observed a single garbled record. Is it possible that there's some kind of protection built in somewhere that "atomizes" my write()s somehow, preventing them from interrupting each other?

FWIW, I'm using Python 3.4.

BTW, in response to some comments below, there's lots of computation in addition to the writing, so I need processes, not threads. And there are upwards of 32 processors, writing 300-400-character text lines. Thanks again for your input, folks.

0 Answers
Related