I am currently trying to parse a big file (~700k lines) with ANTLR using Java Runtime. The file is too big and the parsing leads to an "Out of Memory" exception (after having more than 8GB of RAM assigned).
Since I do not care about parsing everything together, I was able to split the file into several files containing a few functions each. I am now running several threads in order to parse each of the obtained files.
Here is the run() function of my runnable for every thread:
public void run() {
System.out.println("Starting to parse file " + this.filename);
try {
org.antlr.v4.runtime.CharStream stream = CharStreams.fromFileName(this.filename, Charset.defaultCharset());
CPP14Lexer lexer = new CPP14Lexer(stream);
CommonTokenStream tokens = new CommonTokenStream(lexer);
CPP14Parser parser = new CPP14Parser(tokens);
ParseTree tree = parser.translationunit();
lexer = null;
tokens = null;
parser = null;
tree = null;
} catch (IOException e) {
e.printStackTrace();
}
TheParser.current_temp_file.decrementAndGet();
System.out.println("Finished parsing file " + this.filename);
}
Here is my problem: the threads run well. I run them 3 by three (using the "TheParser.current_temp_file" static atomic integer) in order to avoid the memory issue. However, the threads do never completely terminate. Thus, after executing, they are still alive, and the memory keeps on increasing.
I think the problem probably comes from the CharStream, which is never closed. However, there is no such method in the org.antlrv4.runtime.CharStream class.
Do you know where the problem comes from and how I could handle it?
Thank you!