A critical part of every computer is memory. If you go to buy a PC, you will often see how much main memory (4 to 8 Gigabytes is the norm) and hard drive space the machine has. Main memory is implemented using Dynamic Random Access Memory. This is a set of chips on a small board that plugs into the computer. A hard drive is a larger device that plugs in using a standard interface.
These components form the bottom of what is called the memory hierarchy of a machine. In general, this hierarchy goes from fastest access (CPU registers) to slowest (Hard Drive). But, as the memory gets slower, it also can get bigger. A register can hold 128 bytes of information (at the most). Level 1 Caches hold about 32 kilobytes of information. Level 2 caches are bigger, holding two to eight megabytes of information. Main memory is from four gigabytes to sixteen gigabytes for most machines. And hard drives store one to three terabytes.
But, at each step, speed of access goes down. Accessing information on a hard drive is at least a million times slower than L1 cache. A hard drive is tens of thousands times slower than main memory, and main memory is tens of thousands times slower than cache.
One of the main functions of any operating system is to hide this latency (the time it takes to access information) as best it can. But, if you’ve ever booted up your machine, started to work only to hear the hard drive spin and wait, you can see the vast differences in speed that loading information from a hard drive can take.
At the heart of managing memory is the Memory Management Unit. This part of every CPU is responsible for moving information to and from the CPU into main memory. Before information can be used, it must be available in the main memory. To get information from a hard drive, a different part of system is used (this is called the IO subsystem. More on this later).
The tricky part about the MMU is that it gives every program that runs on a computer the illusion of having all the available memory for it’s use. In fact, it provides the illusion of more memory than a machine might have. To understand this, we have to understand the notion of bit size in architecture.
You may have heard about a CPU being 64 bits. Most desktop and laptop processors are 64 bit now, and many smartphone processors are now as well. Fundamentally, the bit size has a lot of impact on the CPU design, but what is useful for us is the impact on what is termed virtual memory.
Virtual memory is the illusion that a MMU provides to a program. For most programs, the operating system gives the program the illusion that it has two gigabytes (sometimes three) of main memory to use. This is because most programs are still targeted to the older 32 bit model for compatibility. 64 bit programs have the illusion that they have one terabyte of main memory. This is way more than any computer can actually provide. The amount of memory a program thinks it has is called the virtual address space.
Basically, all memory is addressable. An address is merely a set of bit (32 bits or 64 bits depending on the program) that points to where a set of information is. The MMU takes these virtual addresses and maps them to the physical memory. As needed, the MMU in cooperation with the operating system moves information to and from main memory to the hard drive. This operation is called paging. Also, the MMU moves information to and from the main memory to the CPU caches. This operation is called caching.
When a program is being run, it is moving millions of bytes of information back and forth from the CPU to memory as billions of instructions are being run. The instructions and data for every program running on a machine need to be loaded into memory. A special program, called the operating system, is responsible for the management and running of other programs and their memory (along with many other tasks). Computer have special hardware (the BIOS) that find and load the operating system from the hard drive.
The operating system uses a complex set of rules to determine what to page in and out of memory. When these rules don’t match how the computer is being used, the computer can slow down considerably. When a computer is not responding, it is often because it is performing a large set of paging operations. In fact, this often referred to as paging. For example, “My desktop is running really slow, it keeps paging all the time”.