mmap with map_ANON for saving large amount data

Viewed 62

I am working on a project which requires to generate a huge amount of data (i.e hundreds of gigabytes) and process them later on. Currently, I am using C++ and I'd like to use mmap() to create some virtual memory to keep the data. The obvious way to do it is to create a file and use mmap() to map let's say 100GB to the file, then we should be able to read/write data from the mapped memory. Something like this:

    int file = open("disk_cache", O_RDWR|O_CREAT, 0644);
    lseek (file,//100GB, SEEK_SET);
    write (file, "", 1);

    auto* mapPtr = (unsigned*)mmap(nullptr, //100GB,  PROT_WRITE|PROT_READ, MAP_SHARED, file, 0);
    if ( mapPtr == MAP_FAILED) {
        perror("mmap");
        return ;
    }
    Generate_data(mapPtr); \\ write/generate data here; 
    Process_data(mapPtr);  \\ read/access data here; 
    if (munmap(mapPtr, //100GB) == -1) {
        perror("mmap:");
        return ;
    }
    close(file);
    if( remove("disk_cache") != 0 ){
       perror("remove the file:");
    }

The above code works fine, but it just looks a bit redundant that I have to create a file and remove it later. I am thinking whether I can MAP_ANON flag to map the data, so I tried somthing like this:

    auto* mapPtr = (unsigned*)mmap(nullptr,  //100GB ,  PROT_WRITE|PROT_READ, MAP_SHARED|MAP_ANON|MAP_NORESERVE, -1, 0);
    if ( mapPtr == MAP_FAILED) {
        perror("mmap");
        return ;
    }

It works fine until the data exceed 64 GB (since I am running this on a server which has 64 GB Mem), My questions are as follows:

(1)It seems like MAP_ANON keeps all the data on Mem, not disk? I wonder why this happened? is there any way to avoid it? I thought MAP_ANON would keep all the data in dev/zero, we can safely store/access data from there.

(2)Also, I am using multiple threads(i.e. 32 cores)to generate/access data. For mmap(), random accessing data is very efficient. But storing/writing data to the memory seems to be slower. I did some quick experiments, it seems like ofstream is faster than mmap() especially when the data is larger than Mem. I feel like maybe I did something wrong with mmap(). Are there any things I need to take care of when using mmap() writing/storing data that are larger than Mem? What's the best way to do it?

Any helps would be appreciated, thanks in advance.

0 Answers
Related