Best way to save a large binary file by chunks asynchronously in C++

user_9132612 0
Viewed 293

I'm developing a C++ application whose output is a single big binary file (a couple of GBs, basically a large sequence of floats). The content of this file is generated asynchronoulsy by parallel processes.

Each time a process finishes, its result has to be saved to its corresponding position inside the binary file in the disk (the order in which processes finish does not necessarily correspond to the order in which their results are to be stored in the disk. It takes about 5 processes to get the full data for the output).

What would be the best way to achieve this in C++? I have a couple solutions that work, but maybe they can be improved in terms of minimizing disk usage:

  • Saving individual files for each finished process, then merging
  • Keeping a fstream open and positioning the put pointer for each save operation using seekp()
disk-io fstream parallel-processing asynchronous c++
1 Answers

I wouldn't recommend wasting time on writing to temp files and merging, if it can be avoided.

Serializing to a single process / single stream will probably be much faster. But make sure to seek-and-write in some chunks of at least 64 KB, to reduce overhead.

I wouldn't use fstreams at all as they come with some overhead (and you're dependent on the quality of implementation as is evident in 1, 2, 3, 4). Better to just use fopen, disable buffering, and write in chunks of 64 KB+.

For even better performance can use memory-mapped I/O, for example using Boost.Iostreams (example). You can memory-map from multiple processes, too.

If the fragments generated by separate processes are a multiple of 4 KB or more, on most OSes you can simply open the same file in each process, seek to the desired location and write (not very portable but OK on Linux, BSD and Win32). On Win32 just need to set file share mode accordingly.

user_4853430
Related