In a Go program I allocate a big array (not a slice):
package main
import "fmt"
var BigArray [10 * 1024 * 1024]byte //10MB
func main() {
fmt.Println(len(BigArray), BigArray[100])
}
When I observe the program's memory footprint, this is not apparent. In fact, no matter how large the array size is. Can someone please explain this?
What you are likely seeing is virtual memory overcommitment rather than any special behaviour on the part of Go. If I compile your test program and inspect the binary with size, we can see that the array hasn't been optimised out, and is represented in the "bss" section of the executable:
$ go build test.go
$ size test
text data bss dec hex filename
1363348 86448 10666480 12116276 b8e134 test
If we increase the size of the array, the bss size sees a similar increase. This section represents zero initialised data in the program. It doesn't affect the size of the executable on disk, since the only thing the OS needs to know is how much of it there is.
When you run the program, the OS reads the executable and loads it into memory so it can be run. A naive implementation of the program loader would allocate the 10MB of memory for the bss data and initialise it to zero. That's not quite what happens on a Linux system.
Instead, it will reserve 10 MB of address space with no backing store. When the program tries to read or write to any memory in that address range, a page fault will be generated. The kernel will respond by allocating a page, initialising it to zero, hook it up to the process's address space, and let the program continue executing.
As this test program only accesses a single item in the array, only a single 4KB page is allocated. The benefit of this system is that the OS only ends up allocating memory that the program actually uses, allowing you to run more programs simultaneously than you otherwise might. The downside is that any page fault could result in an out of memory situation, where the kernel will need to decide to kill one or more processes to continue on.