Resident memory greater than --max-old-space-size threshold?

V8 developer here. The --max-old-space-size flag does not directly control the overall process' memory consumption; it puts a limit on one part (the biggest part) of V8's managed (i.e. garbage-collected) heap, which is the chunk of memory where all the JavaScript objects are put. Aside from this "old space", V8 also has a (much smaller) "new space"; and besides V8's managed heap, there are a bunch of other consumers of memory in the process: for example, within V8, parser and compiler use memory outside the managed heap; and then aside from V8, Node puts a bunch of things in memory too. Depending on what exactly the embedder (i.e. Node) and the executing code do, large strings and ArrayBuffers can also live outside of the managed heap.

In short, --max-old-space-size gives you one knob to affect how much memory will be used, but the limit you set there is not a limit for the process' overall memory consumption. (For comparison, in Chrome, typically about a third of a renderer process' memory is V8's managed heap, though depending on what websites do there are significant outliers in both directions. I don't know typical numbers for Node.)

The RES that you can see on the top is the non-swapped region of the virtual memory of a linux process (see man top for more details). Here is short snipped of the explanation

22. RES  --  Resident Memory Size (KiB)
     A subset of the virtual address space (VIRT) representing the non-swapped 
physical memory a task is currently using.  It is also the sum of the 
RSan, RSfd and RSsh fields.

     It  can  include private anonymous pages, private pages mapped to files 
(including program images and shared libraries) plus shared anonymous pages.  
All such memory is backed by the swap file represented separately under SWAP.

The output of top should not be confused with the node.js memory model elements, since this output is giving the kernel view of the process, while internal memory organizations (like in this case of node.js) are not visible.

If you want to get more details on the structure (the memory map) of the process, you can use pmap -x <PID> and you will see which memory segments are currently in the RES, however this segments are also not to be confused with the node.js segments (although, some of them can be directly mapped to the node.js segments).

node.js memory map, which is the subset of the whole process memory allocated, is called Resident Set and should not be confused with the Resident Memory Size - RES. In fact, the Resident Set of node.js is more closer to the value that you can see in VIRT. The difference between VIRT and the Resident Set will be attributed to the unix shared libraries (not to be confused with node.js modules) that are loaded by the node.js, files that the process had opened, etc.).

The Resident Set is further divided in the Code Segment, Stack Segment and the Heap Segment. The Code Segment, as the name suggests, contains the executable code. The Stack Segment contains the stack informations of the currently running threads in the process. This blog provides good overview of the memory organization.

What remains from all of this is the Heap Segment. And here is where the configuration parameter max-old-space-size has a role to play. The heap of node.js follows memory model that is very similar to Java garbage collection memory model. The parameter is telling the process how much objects are allowed to survive consecutive garbage collection cycles. However, this doesn't mean that this objects are un-swappable, with other words, it's not given that all of the 512MB that are set will reside in RES. This is a subsegment of the Heap. The Old Space is a sub-segment of the Heap.

So, what you read in RES had no direct link to the Old Space Sub-Segment of the Heap Segment. In applications where there is huge garbage left from the short leaving applications contexts (variable is the context that are not retained by the application), the Old Generation will be very small, however the RES will grow rapidly and the setting will not help you to manage the memory of the process.