I need to persistently store content from a large .csv file, that contains the following values:
Lifetime (months);Mileage (km);Percentage
6;5000;72
6;10000;71,5
6;15000;71
.
.
.
6;390000;33,5
6;395000;33
6;400000;32,5
12;5000;70
12;10000;69,5
12;15000;69
.
.
.
12;390000;31,5
12;395000;31
12;400000;30,5
.
.
.
These values are related to the residual value of a car after some period of lifetime and mileage. So I have multiple of these .csv files for different cars with different residual values. Every month I get such a .csv file and every month I need to persist them in the database.
My thoughts:
- Relational database:
It is related to a car, but I think storing this in a relational database table is not efficient. It would generate a unique id for each row (a .csv file can contain > 30.000 rows) for data, which doesn't need a unique id in my opinion. Also most of the columns (lifetime, mileage) would contain repeated values. Furthermore, you could think of having only one row for the entire .csv file in a table, and store the .csv content as a multidimensional array in one cell of the column. But even this is not really efficient I guess, especially for fast reads of a single percentage value.
- File Storage:
I could also store the .csv file directly and have one database with metadata, that reference to the file. But when I want to read one percentage value from the file, I firstly need to read the file from the filesystem, parse it and then search for the value, which is again very inefficient.
- NoSQL:
I think NoSQL has probably similar problems as SQL. I could create one object for each row or store a multidimensional array of the .csv data in one object. But I am not sure with NoSQL, maybe it could be a solution.
- Time-series database:
One interesting point is, that the data is related to the time and the mileage so it is time linear. It is also added each month to the database, but I am not sure if it really fits a time-series database, because it is not a use case such as a stock price every minute for example.
Do you guys have any idea about this problem? I have the feeling that none of the database technologies I know of, fit my use case.