Warning: I am not a security expert or this type of attack.
Cookies and Session do not resolve
Cookies and Session are not suitable places to place this type of security.
Cookies are the responsibility of the client (usually the browser), therefore easily manipulated.
The User Session, although it stays on the server, is just a data structure associated with an identifier, which is usually in a Cookie or in the URL through URL Rewriting .
In the two cases above, requests to perform a brute-force attack simply need to simulate a new user for each request, as if it were someone who had just opened the browser, causing the server to automatically generate a new session.
Security is not just the application
Attacks involving massive network usage such as this or Service Denial (DoS) are generally best handled by Firewalls or Proxies that filter excess requests.
Leaving this in charge of the application will bring complexity to it and will make the team spend time implementing something that is not part of the scope of the system. This distraction almost always comes at the expense of a system with less quality.
And the IP?
A simple but naive technique would be to limit the number of requests using the IP address.
This works in a simple attack, coming from only one source. However, more sophisticated distributed attacks will control multiple sources and it is very difficult to differentiate a "normal" request from an automated one.
How to detect attacks?
Speaking of differentiate , that's the whole point. How can I detect the pattern of artificial requests generated by user-generated attacks?
The answer to this question makes it possible to effectively prevent brute-force attacks.
I will not try to answer this here, as it will be discussed by security experts.
I can even think of little techniques to hinder a hacker's action, but ultimately, it will always be possible for him to identify those techniques and adjust the attack.
Perhaps using a captcha or equivalent "human test" is one of the most effective ways today, however it often disrupts usability and there are also guarantees that an automated algorithm can not be implemented that solves these challenges.
The purpose of this answer is to inform web application developers not to try to reinvent security using "naive" techniques. At the same time, I want to say that security should be a concern, but it requires further study, appropriate tools and expert consultation.