In which cases they return the error Exception of type System.OutOfMemoryException

Basically it is lack of memory. It happens more on 32-bit machines that can have applications of maximum 2GB in total, 64-bit is less common but also has a limit, and it tends to be very slow before this happens.

It may be general lack of memory or it may be that an object is too large. Usually objects can not be more than 2GB, except 64-bit special setup .

Other memory manager issues may cause something like this.

It turns out that this often happens because of the programmer's error. It's not that he's carrying too many large objects in his memory, it's not releasing them.

Many people think that because .NET has a garbage collector can make any mess and everything looks good. Not really. You can not hold memory unintentionally. Memory is only freed when .NET knows that there are no references to that portion of memory and is not an unmanaged resource. If you do this there is a memory leak (it's Java, but it's almost the same).

If you are using a class with the interface Disposable and you are not closing the resource, probably with using will leak.

If you are using an event and do not release the list of signatures when the object is no longer needed, it will leak.

If you put it in a static area or something with a life time close to the lifetime of the application then there is a reference to something that you may not need and have a leak. This may be true for ill-thought algorithms.

There are a number of cases described in the above link that cause too, but these are more common.