I'm working with a java method that reads (string) data from one file, then converts that data into a 2D integer array. A computation is then performed on that 2D array to get a bunch of other related 2D integer arrays which are all stored in one ArrayList. That ArrayList is then looped through and each 2D array is converted to a string and appended to a new text file.
This process is repeated for each 2D integer array that is stored as a string in the original input file.
I had to resort to this kind of implementation because there is simply too much data for the program to store all the 2D integer arrays in ram. Hence my attempted workaround to gather the stored data from a hard drive and perform the necessary computation one 2D array at a time (To not overload the ram and crash the program).
For some more context, there are about 68 Million 2D integer arrays in the original input file (which is about 6Gb worth of txt file). Now I know that for each of these 68 Million 2D integer arrays the program is going to produce 32 other 2D integer arrays thus producing a total of about 2 Billion 2D integer arrays.
My hypothesis is that by the end of the program I should have a txt file that is 6 * 32 = 192Gb in size.
Now I ran this program twice already without altering any of the code and I got two different results that I am struggling to understand. The first time I ran the program I got an output txt file with 68Gb of data whereas the second time resulted in a final txt file of 41Gb.
I checked and confirmed that the final few 2D arrays in the output file were not related to the final 2D array in the input file for both of these attempts.
Here is the code in question:
// This method performs the necessary calculation
public static ArrayList<int[][]> expandEquivalenceClass(int[][] square){
ArrayList<int[][]> solution = new ArrayList<int[][]>();
solution.add(square);
solution = MatrixPermutation.allCycleRotations(solution);
solution = MatrixPermutation.allCycleSwaps(solution);
solution = MatrixPermutation.allD4transformations(solution);
return solution;
}
// This method appends all the related 2D integer arrays to the new file
public static void appendToFile(ArrayList<int[][]> solution, String fileName) throws IOException{
PrintWriter writter = new PrintWriter(new FileWriter(fileName, true));
for(int[][] sol: solution){
for(int[] val: sol){
writter.append(Arrays.toString(val) + "\n");
}
writter.append("\n");
}
writter.close();
}
// This method reads from the input file
// Converts string into a 2D array
// Performs necessary calculations on that 2D array
// Appends the results to the new file
// Reset and repeat for each new 2D array
public static void readFromInitalList(String fileName, int n) throws IOException{
Scanner scanner = new Scanner(new File(fileName));
ArrayList<int[][]> equivalenceClass = new ArrayList<int[][]>();
int[][] square = new int[n][n];
String str = new String();
int count = 0;
while(scanner.hasNextLine()){
if(count == n){
equivalenceClass = expandEquivalenceClass(square);
appendToFile(equivalenceClass, "Z.txt");
equivalenceClass.clear();
count = 0;
}
str = scanner.nextLine();
if(!str.equals("")){
str = str.substring(1, str.length() - 1);
String[] val = str.split(", ");
for(int i = 0; i < n; i++){
square[count][i] = Integer.valueOf(val[i]);
}
count++;
}
}
scanner.close();
}
The readFromInitalList(String fileName, int n) method is called in another java class in another file. I just wanted to see if the community could suggest any ideas as to why all the data won't append to the file or if there are other general improvements that could be suggested.
Would it be wise to attempt to construct the output into files with a size of 10Gb each (ie 20 separate txt files assuming there is a total of 192Gb of solution)?
Any ideas/suggestions/comments are much appreciated - Thank you!