TL;DR - I'm looking for a way to display millions (165M on average) of lines in a grid in C#. There are several questions on this website and elsewhere explaining why this is an insane question, yet I believe I have a use case where this is a legitimate ask, so please read before closing or marking as duplicate.
Full story:
One of our customers has a system that monitors about 100.000 sensors. Each of those sensors logs data to text files which forever keep on growing – one sensor equals one data file. Each line in such a file contains the sensor ID, a timestamp, some parameters, and the sensor data itself, always with a maximum length of 80 characters.
They have a very old piece of software running in a DOS like environment where they can choose one sensor, and scroll through the text file.
When they initially contacted us for a solution we thought this would be a “simple” task: read the sensor data into a database, display a subset of the data in a grid in which they could filter on all parameters, job done.
Turns out they don’t “read” the data – they scroll through those lists holding down the down or pagedown key and search for patterns in the sensor data. We’ve asked to define those patterns so we could suggest to do the searching in software, but the researchers told us that these patterns aren’t fixed, and can change depending on what they are looking for.
I’ve had a meeting with a few of the researchers and the only decent comparison I can make is that it’s like watching someone look at The Matrix – they scroll through lines of gibberish yet see the data they are looking for when they see it (without exactly knowing what they are looking for).
While this seems like a very strange way of working, it has to be said that the DOS environment they have now is extremely efficient – sensor logs vary from a few 100k lines to extremes of 4000M lines (averaging 165M lines), yet scrolling through them is smooth and without a single hiccup.
I have no idea how they accomplished this back then, but now I’m tasked with recreating the same setup for modern machines in C#, hence my question. The only thing I know about the old application is that it was written in assembler, and that it doesn’t scroll line-per-line – it scrolls pixelline per pixelline to make for a very smooth scrolling interface.
So – how can I display millions of lines in a grid in C#, and make sure the user can scroll through the list (both up and down) without any load times or hiccups?
Thanks!