I am working on a project which requires to generate a huge amount of data (i.e hundreds of gigabytes) and process them later on. Currently, I am using C++ and I'd like to use mmap() to create some virtual memory to keep the data. The obvious way to do it is to create a file and use mmap() to map let's say 100GB to the file, then we should be able to read/write data from the mapped memory. Something like this:
int file = open("disk_cache", O_RDWR|O_CREAT, 0644);
lseek (file,//100GB, SEEK_SET);
write (file, "", 1);
auto* mapPtr = (unsigned*)mmap(nullptr, //100GB, PROT_WRITE|PROT_READ, MAP_SHARED, file, 0);
if ( mapPtr == MAP_FAILED) {
perror("mmap");
return ;
}
Generate_data(mapPtr); \\ write/generate data here;
Process_data(mapPtr); \\ read/access data here;
if (munmap(mapPtr, //100GB) == -1) {
perror("mmap:");
return ;
}
close(file);
if( remove("disk_cache") != 0 ){
perror("remove the file:");
}
The above code works fine, but it just looks a bit redundant that I have to create a file and remove it later. I am thinking whether I can MAP_ANON flag to map the data, so I tried somthing like this:
auto* mapPtr = (unsigned*)mmap(nullptr, //100GB , PROT_WRITE|PROT_READ, MAP_SHARED|MAP_ANON|MAP_NORESERVE, -1, 0);
if ( mapPtr == MAP_FAILED) {
perror("mmap");
return ;
}
It works fine until the data exceed 64 GB (since I am running this on a server which has 64 GB Mem), My questions are as follows:
(1)It seems like MAP_ANON keeps all the data on Mem, not disk? I wonder why this happened? is there any way to avoid it? I thought MAP_ANON would keep all the data in dev/zero, we can safely store/access data from there.
(2)Also, I am using multiple threads(i.e. 32 cores)to generate/access data. For mmap(), random accessing data is very efficient. But storing/writing data to the memory seems to be slower. I did some quick experiments, it seems like ofstream is faster than mmap() especially when the data is larger than Mem. I feel like maybe I did something wrong with mmap(). Are there any things I need to take care of when using mmap() writing/storing data that are larger than Mem? What's the best way to do it?
Any helps would be appreciated, thanks in advance.