Can Go work with memory at the page level?

user_6327846 0
Viewed 159

I am a Go developer reading a book called Database Internals. The author talks extensively about working with memory at a very low level (specifically, at the page level).

As I am trying to build my own database, I have looked through Go documentation and other articles for discussions about work with memory at this level, however nothings seems to talk about working with data in a specific page of memory.

My questions are:

Can Go do this?

If Go can't do this, is this something that C or C++ can do?

If none of Go, C, or C++ can do that, perhaps I'm not understanding what the author means by working with memory at the page level, how should I be thinking about it?

If necessary, please let me know of any further detail that needs to added to the question.

Excerpt from the book

Context is that we're talking about implementing B-Trees and grouping nodes in pages. This is found in chapter 2, section titled "On-Disk Structures"

PAGED BINARY TREES Laying out a binary tree by grouping nodes into pages, as Figure 2-6 shows, improves the situation with locality. To find the next node, it’s only necessary to follow a pointer in an already fetched page. However, there’s still some overhead incurred by the nodes and pointers between them. Laying the structure out on disk and its further maintenance are nontrivial endeavors, especially if keys and values are not presorted and added in random order. Balancing requires page reorganization, which in turn causes pointer updates.

It also provides this image: enter image description here

go memory c c++
2 Answers

The book is talking about "On-Disk Structures". Page in this context simply means a block of data. Disk access works in sectors or clusters, so the data should be optimized for locality in order to fit in those blocks as best as possible. The challenge in database file design is to translate data's temporal locality into spatial locality.

But the concept can apply to RAM, too. Working at page level in user-space virtual memory simply means being aware of the underlying memory architecture, and optimizing for it.

On x86, user-space virtual memory is organized in pages of 4 KB*.

That means that when working with large amounts of data, it can be worthwhile to work with memory regions which are a multiple of 4 KB, aligned to 4 KB.

In C there are many ways to achieve that, e.g. aligned_alloc.

But Go goes further - it already aligns large arrays at least on page size (on Mac, BSD, Linux and Windows).

func main() {
    buf := make([]byte, 1024*1024)
    fmt.Printf("%p\n", unsafe.Pointer(&buf[0]))
}

Will print something like:

0xc000180000

And in case you encounter an implementation that doesn't, you can always align a byte slice "manually" by allocating PageSize-1 extra bytes and then skipping the bytes at the beginning which do not start at a multiple of PageSize, using the alignment formula:

offset = (alignment - base) & (alignment - 1)

func main() {
    buf := make([]byte, 1024*1024 + 4096 - 1)
    base := unsafe.Pointer(&buf[0])
    offset := (4096 - uintptr(base)) & (4096 - 1)
    aligned := buf[offset:]
    fmt.Printf("base   : %p\n", unsafe.Pointer(&buf[0]))
    fmt.Printf("aligned: %p\n", unsafe.Pointer(&aligned[0]))
}

(can't think of a platform where the above would print different values)

* Some platforms have large page support for 2MB and 1GB pages. Go uses those automatically when available.

user_4853430

I might be missing something, but working with Physical memory (in contrary to Virtual memory) is not a Language feature, but of the OS.
So the answer to your question is: Yes, Go can do this too.

Each operating system provides methods to work with the actual address space, These services require elevated permissions and come as Driver/Kernel-Module and using this access for Memory Mapped IO or pre-allocating block of physical memory to work with (before the MMI uses it as paged memory).

I guess pre-allocating memory block on system load (by driver) and working with that is what you wish to do.

If you intend to access "Virtual Memory" on the "Physical Memory" I'd discourage you and say that it is an extremely delicate and fragile mechanics and unless you find a HUGE advantage for doing so, I cant see any justification of going down this road.

user_6483680
Related