Memory profiling with data.table

user_1174421 0
Viewed 397

What is the correct way to profile memory in R code that contains calls to data.table functions? Let's say I want to determine the maximum memory usage during an expression.

This reference indicates that Rprofmem may not be the right choice: https://cran.r-project.org/web/packages/profmem/vignettes/profmem.html

All memory allocations that are done via the native allocVector3() part of R's native API are logged, which means that nearly all memory allocations are logged. Any objects allocated this way are automatically deallocated by R's garbage collector at some point. Garbage collection events are not logged by profmem(). Allocations not logged are those done by non-R native libraries or R packages that use native code Calloc() / Free() for internal objects. Such objects are not handled by the R garbage collector.

The data.table source code contains plenty of calls to Calloc() and malloc() so this suggests that Rprofmem will not measure all memory allocated by data.table functions. If Rprofmem is not the right tool, how come Matthew Dowle uses it here: R: loop over columns in data.table?

I've found a reference suggesting similar potential issues for gc() (which can be used to measure maximum memory usage between two calls to gc()): https://r.789695.n4.nabble.com/Determining-the-maximum-memory-usage-of-a-function-td4669977.html

gc() is a good start. Call gc(reset = TRUE) before and gc() after your task, and you will see the maximum extra memory used by R in the interim. (This does not include memory malloced by compiled code, which is much harder to measure as it gets re-used.)

Nothing I've found suggests that similar issues exist with Rprof(memory.profiling=TRUE). Does this mean that the Rprof approach will work for data.table even though it doesn't always use the R API to allocate memory?

If Rprof(memory.profiling=TRUE) in fact is not the right tool for the job, what is?

Would ssh.utils::mem.usage work?

data.table memory-profiling r
2 Answers

This is not related to data.table. Recently there was a discussion on twitter about same dplyr behaviour: https://mobile.twitter.com/healthandstats/status/1182840075001819136

/usr/bin/time -v Rscript -e 'library(data.table); CJ(1:1e4, 1:1e4)' |& grep resident

There is also interesting cgmemtime project, but it requires a little bit more setup.

If you are on Windows I suggest you to move to Linux.

user_24904970

If you are using Windows, you can call Powershell memory and other performance objects for RGui and Memory Compression as system commands through R and call various memory counters. I don't have a path to store Powershell objects in R yet. Powershell Code for RGui and 'Memory Compression' which Windows uses to store frequently used objects:

    $t1 = ps | where {$_.Name -EQ 'RGui' -or $_.Name -EQ 'Memory Compression'};
    $t2 = $t1 | Select { $_.Id;
    [math]::Round($_.WorkingSet64/1MB);
    [math]::Round($_.PrivateMemorySize64/1MB);
    [math]::Round($_.VirtualMemorySize64/1MB) };
    $t2 | ft * 

    $t1 | gm -View All
    $t1.Modules
    $t1.MaxWorkingSet

Powershell embedded in R:

    ps_f <- function() { system("powershell -ExecutionPolicy Bypass -command $t1 = ps | where {$_.Name -EQ 'RGui' -or $_.Name -EQ 'Memory Compression'};
    $t2 = $t1 | Select { 
     $_.Id;
     [math]::Round($_.WorkingSet64/1MB);
     [math]::Round($_.PrivateMemorySize64/1MB);
     [math]::Round($_.VirtualMemorySize64/1MB) };
    $t2 | ft * "); }

    ps_f()

     $_.Id;                                                                                                                
     [math]::Round($_.WorkingSet64/1MB);                                                                                   
     [math]::Round($_.PrivateMemorySize64/1MB);                                                                            
     [math]::Round($_.VirtualMemorySize64/1MB)                                                                             
    -----------------------------------------------------------------------------------------------------------------------
    {2264, 1076, 3, 1401}                                                                                                  
    {15832, 3544, 6691, 11965}   



    ps_mem <- function() { system("powershell -ExecutionPolicy Bypass -command $t1 = ps | where {$_.Name -EQ 'RGui' -or $_.Name -EQ 'Memory Compression'};
    $t1 | Select ProcessName,MaxWorkingSet,MinWorkingSet,PagedMemorySize64,NonpagedSystemMemorySize64;")} 

    > ps_mem()

    ProcessName                : Memory Compression
    MaxWorkingSet              : 
    MinWorkingSet              : 
    PagedMemorySize64          : 3411968
    NonpagedSystemMemorySize64 : 0

    ProcessName                : Rgui
    MaxWorkingSet              : 1413120
    MinWorkingSet              : 204800
    PagedMemorySize64          : 7014719488
    NonpagedSystemMemorySize64 : 6662736

    # run some data.table operation

    > ps_mem()
    ProcessName                : Memory Compression
    MaxWorkingSet              : 
    MinWorkingSet              : 
    PagedMemorySize64          : 3411968
    NonpagedSystemMemorySize64 : 0

    ProcessName                : Rgui
    MaxWorkingSet              : 1413120
    MinWorkingSet              : 204800
    PagedMemorySize64          : 7015915520
    NonpagedSystemMemorySize64 : 6662736

Powershell Code: 

    $t1 | where {$_.ProcessName -eq "Rgui"} | Measure-Object -Maximum *memory* | ft  Property,Maximum

Powershell embedded in R:

    ps_mem_ <- function() { system("powershell -ExecutionPolicy Bypass -command $t1 = ps | where {$_.Name -EQ 'RGui' -or $_.Name -EQ 'Memory Compression'};
    $t2 = $t1 | where {$_.ProcessName -eq 'Rgui'}; 
    $t2 | Measure-Object -Maximum *memory* | ft  Property,Maximum ")} 

    # having some problems with rollover...

    > ps_mem_()

    Property                       Maximum
    --------                       -------
    NonpagedSystemMemorySize       6662736
    NonpagedSystemMemorySize64     6662736
    PagedMemorySize            -1570734080
    PagedMemorySize64           7019200512
    PagedSystemMemorySize           680240
    PagedSystemMemorySize64         680240
    PeakPagedMemorySize        -1260961792
    PeakPagedMemorySize64      11623940096
    PeakVirtualMemorySize       -161009664
    PeakVirtualMemorySize64    17018859520
    PrivateMemorySize          -1570734080
    PrivateMemorySize64         7019200512
    VirtualMemorySize           -339103744
    VirtualMemorySize64        12545798144

    some data.table run

    > ps_mem_()

    Property                       Maximum
    --------                       -------
    NonpagedSystemMemorySize       6662736
    NonpagedSystemMemorySize64     6662736
    PagedMemorySize            -1570734080
    PagedMemorySize64           7019200512
    PagedSystemMemorySize           680240
    PagedSystemMemorySize64         680240
    PeakPagedMemorySize        -1260961792
    PeakPagedMemorySize64      11623940096
    PeakVirtualMemorySize       -161009664
    PeakVirtualMemorySize64    17018859520
    PrivateMemorySize          -1570734080
    PrivateMemorySize64         7019200512
    VirtualMemorySize           -339103744
    VirtualMemorySize64        12545798144

To see all the Rgui objects: 

    $t1 | gm -View All


       TypeName: System.Diagnostics.Process

    Name                       MemberType     Definition
    ----                       ----------     ----------
    Handles                    AliasProperty  Handles = Handlecount
    Name                       AliasProperty  Name = ProcessName
    NPM                        AliasProperty  NPM = NonpagedSystemMemorySize64
    PM                         AliasProperty  PM = PagedMemorySize64
    SI                         AliasProperty  SI = SessionId
    VM                         AliasProperty  VM = VirtualMemorySize64
    WS                         AliasProperty  WS = WorkingSet64
    Disposed                   Event          System.EventHandler Disposed(System.Object, System.EventArgs)
    ErrorDataReceived          Event          System.Diagnostics.DataReceivedEventHandler ErrorDataReceived(System.Object, System.Diagnostics.DataReceivedEventArgs)
    ...
user_3056750
Related