I want to write log file, unstructured format (one line at a time), using mmap (for speed). What is the best procedure? Do I open empty file, truncate to 1 page size (write empty string to resize file?), then mmap - and repeat when mmaped area full?
I usually use mmap for writing fixed size structures, usually just one page at a time, however this is for writing log files (anywhere from 0.5 - 10 Gb) using mmap but not sure what's the best practice once the first mmaped area is filled - munmap, resize file truncate and mmap next page ?
While writing logs to memory area, I would tracking size, and msync , what is the proper handling once I get to the end of the mapped memory area?
Let's say I never need to go back or overwrite existing data, so I only write new data to file.
Q1: When I get to the end of mapped area do I munmap, ftruncate file to resize by another page size and mmap the next page ?
Q2: Is there a standard way to pre-empt and have the next page ready in memory for next write? Do this on another thread when we get close to the end of mapped area ?
Q3: Do I madvise for sequential access?
This is for real time data processing with requirement to keep log file - currently I just write to file. Log file is unstructured, text format, line based.
This is for linux/c++/c optionally testing on Mac (so no remap [?]).
Any links/pointers to best practices appreciated.