I have a massive data structure that needs to load, effectively, instantly. It's many terabytes of data - a graph, in fact. A buttload of nodes with links to other nodes. This data structure needs to be baked in such a way that I can stand up a service can access any part of it without a multi-millisecond startup delay or overburdensome per-access times.
In the C++ world, I'd pre-bake the data structure into a file. There are tricks that would then allow me to map that file to a specific virtual address, allowing me to even pre-bake pointers (though, it is possible to replace pointers with offsets and map wherever the OS put it.) I'd then memmap the file and the load would be effectively done. Everything would be instantly available the same way a swapped-out page is.
Java doesn't make it as easy. At best, I could mmap the file, then look up nodes Java data structures in the virtual machine and, having failed to find them there, load them on-demand from the map. I'm not as happy with this solution as it will generate a lot more swapping. It would also require more memory if nodes' sizes are non-constant.
In short, if I needed to pack a many-TB+ sized data structure in a file in a way that it could be directly referenced in Java, how would I do it?