The Memory Layout of a 64-bit Linux Process

The precesses memory space layout is platform specific. On current x86_64 CPU’s the memory will be laid out according to "virtual memory map with 4 level page tables" which is specified in the Linux kernel documentation under Documentation/x86/x86_64/mm.txt:

As you can see, this gives us a virtual address space of 48 bits where 47 bits can be used for user space programs, to address at most 128 TB of memory. Because this article focuses on the user space, I’ve omitted most of the predefined kernel space regions. It’s just interesting to see ➊, that the kernel maps the complete physical memory into the kernel address space for convenience. Notice, that current x86_64 CPUs can only address a maximum of 64 TB because they limit the physical addresses to 46 bits (i.e. a program can use a virtual address space of 128 TB but only 64 TB out of the 128 can be physically mapped). Future versions of x86_64 CPUs will be able to use 57 bits for the virtual address space (resulting in a 56 bits or 128 PiB user space) and up to 52 bits for physical addresses (resulting in up to 4 PiB of physical memory). This new hardware generation requires an extended, 5-level page table as described in Intel’s "5-Level Paging and 5-Level EPT" white paper and recently implemented in the Linux 4.12 kernel. The resulting, new virtual memory map is described in the previously mentioned mm.txt kernel documentation file as well.

As a baseline example we take a slightly modified "Hello world" example as shown in Listing Listing 1. We simply add a call to getchar() at the end of the program ➊ such that we can easily analyze its memory layout.

As demonstrated in Listing Listing 2, we will use the pmap utility to display the memory layout of a process. pmap is actually just a front-end for the memory map of a process as recorded in the smaps file of the proc file system.

If called only with a PID argument, the output of pmap will be more compact and only contain the Address, Perm, Size and Mapping columns. If called with -X or -XX, it will display more of / all off the information exposed by the kernel through /proc/PID/smaps. This content may vary across kernel versions. In our concrete example (with -X) the Address column contains the start address of a mapping. Perm displays the permission bits (r = read, w = write, x = execute, p/s = private/shared mapping). Offset contains the offset into the mapped file and will be zero for all non-file (i.e. anonymous mappings). The same holds true for the Device and Inode columns which are relevant for file mappings only. The Size column contains the size of the memory mapping in kilobytes (notice however that the kernel can only map memory in chunks which are a multiple of the "kernel page size" - normally 4 KB on x86). The RSS ("Resident Set Size") column displays the amount of memory which is actually in RAM while the PSS ("Proportional Set Size") column is the amount of memory in RAM divided by the number of processes sharing it.

As you can see, all the lines excluding the last one describe memory regions in user space. Notice that the last page of the user space (i.e. 0x7ffffffff000-0x7ffffffff000 ➊) will not be displayed by pmap nor is it recorded in the corresponding /proc/<pid>/smaps file. As described in the kernel sources, this special guard page (i.e. 4 KB) is an implementation detail only required on x86_64 CPUs (and included here for completeness).

The last line (i.e. 0xffffffffff600000…​[vsyscall] ➋) refers to a 4 KB page in the kernel space which has been mapped into the user space. It is relict of an early and now deprecated implementation of so called "virtual system calls". Virtual system calls are used to speed up the implementation of certain system calls like gettimeofday() which only need read access to certain kernel variables. The idea is to implement these system calls such that they don’t switch into kernel mode because the kernel maps the required data read-only into the user space memory. This is exactly what the [vsyscall] mapping is used for: it contains the code and the variables for the mentioned virtual system calls.

Unfortunately the size of the [vsyscall] region is restricted to 4 KB and it’s location is fixed which is considered an unnecessarily security risk in the meantime. The vsyscall implementation has therefore been deprecated in favour of the new vDSO (i.e. "virtual Dynamic Shared Object") implementation. This is a small shared library which is exported by the kernel (see vdso(7)) and mapped into the user address space ➌. With vDSO the kernel variables are mapped into an extra, read-only memory region called [vvar] ➍. Both these regions have no size limitations and are subject to ASLR.

Files are read by the Linux kernel with a certain read-ahead which is device dependent and which can be configured with the blockdev(8) command.

According to the man page, the output is in 512-byte sectors so a value of 256 means a read-ahead of 128 KB.

Let’s now start our journey with the execution of the standard C-library function execve() which in turn executes the execve system call. execve is usually called right after fork() and replaces (i.e. overlays) the old programs stack, heap and data segments with the ones of the new program (see the man page of execve(2)).