What is "overlay" and what is its connection with memory?

This is used on primitive operating systems. The famous MS-DOS used this technique. Today it is only used on very limited devices, though some are becoming quite popular with IoT . In general the operating system does not have this because of lack of space to create the mechanism of virtual memory and mainly because it is made to run in hardware that does not provide facilities to manage the memory of simple form, transparent and with performance.

Think of the DLL. It's a way to break an executable into parts. The overlay is the same thing, it's only done in a slightly different way. In fact one of the advantages is to break the executable into parts to be distributed separately, in the past more useful than today since the diskettes had very little space and was the most used means of transportation.

The mechanism is useful when the executable is too large to fit in memory. It splits the code into parts that can be loaded alternately as needed. There is usually code insertion to manage the need to load another piece of code that is not already in memory.

In more complete operating systems in modern architectures there is the virtual memory system, so it does not matter if what you need is in physical memory or not, suffice it to say that it is in memory that exists beyond the limits of RAM, usually a part may be on disk. The application does not need to know anything of this, for it exists a huge memory limited in 4GB in 32 bits or as much as the OS allows until the limit of 16EB in 64 bits. The OS manages whether to put a part in the secondary storage or keep it in RAM.

The Virtual Memory is a management that determines where memory pages (commonly 4KB blocks) are in memory. These pages may be scattered throughout the RAM or may be elsewhere . In theory it may even be on another machine. It controls this, it's not a problem for the user or application developer. It's an abstraction to make a lot of things easier, even to protect areas of memory and to consistently address memory allocation requests for applications.

In general, a modern executable is usually loaded with a technique of memory-mapped file , which makes it transparent if on disk or RAM. Physically, an excerpt of the executable must be in RAM to be executed, so every time you try to access it and can not do it, a page fault occurs (Wikipedia) and the OS will bring the page that should be on disk to RAM . So a code or data being used in two different processes does not have to be duplicated, physically it only exists once despite being referenced in two virtual memory spaces. This is why it is complicated to measure application memory consumption, there is the physical consumption of RAM, total consumption and virtual consumption.

overlay is a poor virtual memory:)