I'm having a problem with an application I'm developing where I have a database with approximately 20 million lines.
The problem is that the user wants to see all 20 million rows in a single screen, even if the scroll bar is huge, this table was formerly exported to Excel and it used Excel for this but as in Excel there is a limitation of approximately 1 million lines they want this application to be developed.Any suggestions on how to do this so you do not miss out on such performance?
Notify the user that it is not possible to load or display "everything on a single screen."
By making a rough estimate, if each bank record took an average of 512 bytes, it would need approximately 9.5 Gigabytes of memory just to store the data on that screen.
Note that much more memory would be required to render these elements in some visual component. Of course it is possible to use some paging technique in memory, but still the performance would suffer.
Add to that the need to transfer almost 10GB over the network to load the screen completely. That would be almost equivalent to downloading a double-sided DVD!
What you could do is simulate or give the impression that all the data is there.
You can use some asynchronous paging scheme where the user selects the page or some filter that limits the data that he wants to see and then you only load enough records to populate the screen.
If your query is efficient, the screen will load almost immediately, but there will be a small delay between each navigation. Something much more tolerable.
I think the question is not about performance, but about usability.
Generally, it's worthwhile to implement a good search engine than doing what you're proposing.
Committed Usability
Imagine a scroll bar 1000 pixels high ( almost the height of a 1600 x 1200 monitor ). This means that for every pointable pixel, there will be 20.000.000 / 1.000 records = > % with% records per pixel.
It turns out that by scrolling through the logs, you will go to the first of that 20,000 sequence, you will always show how many will fit on the screen, and the rest of the 20000 is hidden ... here comes the question: How are you going to do to access the rest of the 20,000 records? With the arrow on the keyboard? With the mouse scroll wheel? Page down and page up keys. This is unfeasible. What's more, hunting for pixels on the scrollbar is already somewhat counterproductive to start the conversation.
Alternative:
User is usually like this, does not know what he wants ... he thinks: I need access to any record, so I put all the records on the same screen. He does not think, I need access to any record, so I need an effective way to reach them, so I need a search engine that meets my needs.
