Why you should forget about 4GiB of RAM on 32-bit systems and move on

In my previous post on 64-bit systems I was somewhat dismissive of applications being able to use PAE and other technologies to allow a 32-bit operating system to use more than 4GiB of memory.  We’ve since asked:

If my operating system is 64 bit can my 32 bit applications use 64 bit memory (> 3.5GB)?

And the answer is, in truth, yes.  Yes, you can.  But it’ll cost you and, probably, make your life difficult.  It’ll also make people wonder why you wanted to try it in the first place too, what with all these new-fangled 64-bit processors and operating systems flying about using whatever memory they need to do whatever they like.

There are many technologies for achieving this, these include PAE, PSE, PSE-36, AWE (Windows), mmap (Unix/Linux) and they all have one thing in common: they require special support to use properly. In some cases forcing the issue means things can and will break, horribly.

TL;DR Version: All these technologies are there for developers. They provide no immediate solution for end users.

To begin with I should also mention that PAE/PSE/PSE-36 are features that allow a 32-bit OS to see more than 4GiB of RAM.  AWE and MMAP actually allow more memory to be used by a 32-bit application on both 32- and 64-bit Operating Systems, even if in a slightly odd way.

PAE, PSE, PSE-36

These are all processor features that basically allow the operating system to see and make use of  more than 4GiB of memory and are not usable by userland programs.  The main reason for this is that the memory that the operating system kernel presents to applications is not the same thing as real, physical memory, and hasn’t been for a long time.  The kernel protects applications from each other by supplying each and every application with it’s own imaginary address space and maps devices and memory requests into that address space. If an application makes a memory request then the operating system and CPU translate that memory request into a real memory address and return the result transparently.  The memory that one application allocates in it’s own address space is not seen in the address space of another application.

This is what operating systems call “Virtual Memory” and it is this virtual memory that is allocated by an application, the operating system then uses both physical memory and hard drive swap files to fulfil and store the data from these the memory requests.  Operating system engineers tend to get annoyed when users talk about turning off the swap file as “disabling virtual memory” as there is no way you can disable virtual memory in a modern operating system and both the swap file and physical memory are simply storage devices that the virtual memory system uses to store data.

The kernel has to be aware of all the memory in the system and, via PAE and other methods, it can be.  The processor itself can have more than 32-address lines connecting it to memory in order to access more memory but the width of a usable register (and therefore address) is still 32-bit and so special addressing schemes such as PAE are used.  From Wikipedia’s entry on PAE:

x86 processor hardware-architecture is augmented with additional address lines used to select the additional memory, so physical address size increases from 32 bits to 36 bits. This, theoretically, increases maximum physical memory size from 4 GiB to 64 GiB. The 32-bit size of the virtual address is not changed, so regular application software continues to use instructions with 32-bit addresses and (in a flat memory model) is limited to 4 gigabytes of virtual address space. The operating system uses page tables to map this 4-GiB address space into the 64 GiB of physical memory. The mapping is typically applied differently for each process. In this way, the extra memory is useful even though no single regular application can access it all simultaneously.

The operating system can effectively use different page tables to switch the processor to a different area of memory, run the application there for a while, then switch to another bank of memory (page table) and run that application and so on.

The problems with PAE can arise when drivers that are not PAE aware inject themselves into or reside in kernel space, they have to be aware that whatever memory they use either has to be mapped into every application page table or be aware that what they are accessing may not be in the current page table, I would assume the operating system will ensure that the driver itself is in every page table but from there onwards the driver is expected to do the work.  Due to this a badly written driver that assumes a single flat 4GiB memory space could allocate memory while one process (and therefore one particular page table) is in use then, not knowing that the page table has been switched, try to read from or write to that memory.

Voila, instant Bluescreen as Windows reports a driver tried to use memory that is not allocated in that page table.

PSE (and it’s evolution PSE-36) is a another related method for the operating system to change the page tables in order for the operating system to be able to map more memory into it’s own address space, and is effectively bound by the same 32-bit application virtual address space rules as PAE.  The extra memory is still only usable by the operating system.

But Windows supports PAE….

Microsoft Windows operating systems have supported PAE for some time now and so, in theory, could support much more than just 4GiB of memory.  Microsoft chose not to do this for compatibility reasons.

Due to the problem I outlined above with PAE and page table remapping the process of fully enabling all of the features that PAE allows means that every driver for every device absolutely has to be recompiled to support PAE.

Microsoft instead chose to enable parts of PAE because it then allows access to the NX Bit which improves computer security, but not to use the full available address space as it would break almost every current driver used in Windows. So for reasons of backwards compatibility they stuck with the flat 4GiB memory space for consumer versions of Windows.  It would require every hardware manufacturer to recompile PAE drivers for almost every device in the last 10 years to fix the problem.

The move to 64-bit breaks everything in a similar fashion, but at least the end user gets a 64-bit address space out of the deal.

Linux and it’s siblings are in a different boat as all the source for kernel, network card drivers and so on are almost universally freely available (with the exception of dedicated graphics card drivers) and are effectively recompiled every time a new version of the distribution is made.  Users can even recompile the kernel and related driver modules themselves if they see fit.  Microsoft does not supply the source to everything in the way Linux does and this is why it can not just simply have everything recompiled to support PAE.

AWE & MMAP

In the same way that the operating system can switch page tables for an entire application and switch particular parts of physical memory into an applications address space Address Windowing Extensions (AWE) allows a an application to map “windows” of physical memory into it’s own virtual address space. By allocating and swapping where these “windows” point to an AWE aware process would be able to make use of more that 4GiB of memory, subject to the limitation that not all of it will be available at any given moment.  The application would have to do a lot of work to keep track of what memory “windows” are storing what information and try to make the best choices as to which it wants to use at any given time.  Microsoft has this to say:

AWE does not require PAE … but is often used together with PAE to allocate more than 4 GB of physical memory from a single 32-bit process.

mmap() is a Linux/Unix API for mapping files into your address space (subject to the repeatedly mentioned 32-bit virtual address space limitations) and apparently you can achieve something similar to Windows’ AWE system using tmpfs:

All “extra” (>4GB) space can be used like a file in a filesystem, not like a plain memory. Think of read()/write() (or pread()/pwrite() for that matter), but much faster ones compared to disk-based storage — in tmpfs. You can also mmap() *parts* of such a file, but will be still limited to 4GB at once — in order to have more, you will have to unmap() something.

As you can probably imagine for very, very large datasets all this juggling of memory could incur quite a substantial performance penalty.

What does it all mean?

You see, all these alternatives are at best workarounds and they require the programmer to work hard on support for them and then also force them to cater for the scenarios where they may not be available.  Typically the only people willing to invest time and effort writing a program to support these technologies are ones who either have a large amounts of time and money to throw at the problem and are limited to a 32-bit OS for whatever reason.  Large database servers seem to be the area where this happened most, as evidenced by Microsoft’s support for large memory configurations:

The reason why 32-bit datacenter versions get large memory support is because their hardware tends to have better support, and the customers are willing to pay to get that support.

I honestly can’t see a reason for any new application to be using these features, if you truly need more than 4GiB of memory at a time then 64-bit is the fastest and by far the easiest way to go.  Granted your old 32-bit applications are stuck with 4GiB limitations but, as always, “out with the old and in with the new.”