In one of the technical discussion, I was asked a solution for the below scenario without using any standard java libraries which handles similar situation
- how do you process an array which does not fit in available memory for JVM
- how do you process a big file say 20 gb which does not fit in available memory.
I can possibly think of the following solutions
- get the length of the array and process the array using some part of the length(for ex 4 iterations on array with length/4)
- original file can be split into multiple parts using split command (or something similar in respective OS environment). Process individual smaller files and generate the intermediate result (for example data aggregation). Once individual file processing is done then based on the final size of the intermediate result files, process all the result file in 1 go or again apply iterative processing.
However I would like to know if there is a better approach. Also if such questions are inappropriate for this forum then please let me know and I will delete the question. While I found some google articles with a solution but it talks of solution already provided by some libraries and hence posting it here.