So I am running some code which runs over 300k times. Each time this code runs, it returns up to 300k values. I am currently storing the results I get in an ArrayList:
List<List<Object>> thisList = new ArrayList<List<Object>();
for (int i = 0; i < 300000; i++) {
thisList.add(new ArrayList<Object>());
}
for (int i = 0; i < 300000; i++) {
List<Object> result = someCode();
for (Object obj : result) {
thisList.get(obj.id).add(obj.value);
}
}
In this code, everytime obj is obtained, it has a value obj.id which specifies the index in the List where obj.value has to be stored.
What would be the most efficient way to store the results elsewhere as the search continues? My code seems to stop working past iteration 400, most likely due to low memory issues. I have considered using a simple text document where each line represents a List<Object> but through some Googling, it seems there is no way to append to a specific line, and all suggestions seems to point towards overwriting the entire text document. I've never worked with databases before which is why I am trying to avoid that for now.
Would appreciate if someone can give me suggestions on what I could do.
Edit: Is there a method which does not use a database, where after each iteration of the outer for loop, the data can be stored?
For example, given a file which currently contains
List 0: obj.value1 obj.value2
List 1: obj.value1 obj.value4
...
List 300000: obj.value3 obj.value8
and result contains
{obj<1, 100>, obj<0, 3>, ...}
where each object is of the form obj<id, value>, the file becomes
List 0: obj.value1 obj.value2 obj.value3
List 1: obj.value1 obj.value4 obj.value100
...
List 300000: obj.value3 obj.value8