Firrtl runs of out of heap memory with large input

user_6536722 0
Viewed 401

I'm attempting to run a verilog compiler pass on a 110MB Firrtl file and I'm consistently getting out of memory errors, despite giving it a roomy 12G heap space to play with. It seems like the problem is the parser, specifically ANTLR, making too many ArrayLists. Is 110MB considered far too large for a Firrtl circuit definition?

The file was produced with Chisel via Firrtl with --compiler=high. My Firrtl version is 1.2-SNAPSHOT (built locally) and Chisel is 3.2-SNAPSHOT (also built locally).

$ JAVA_OPTS="-Xms8G -Xmx12G" ../firrtl/utils/bin/firrtl -i mnist_cnn_v4.fir -o mnist_cnn_v4.v -X verilog --no-check-comb-loops --no-dce --info-mode=ignore -ll Trace
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3688)
    at java.base/java.util.ArrayList.grow(ArrayList.java:236)
    at java.base/java.util.ArrayList.grow(ArrayList.java:241)
    at java.base/java.util.ArrayList.add(ArrayList.java:466)
    at java.base/java.util.ArrayList.add(ArrayList.java:479)
    at org.antlr.v4.runtime.ParserRuleContext.addAnyChild(ParserRuleContext.java:134)
    at org.antlr.v4.runtime.ParserRuleContext.addChild(ParserRuleContext.java:145)
    at org.antlr.v4.runtime.Parser.consume(Parser.java:584)
    at firrtl.antlr.FIRRTLParser.intLit(FIRRTLParser.java:2870)
    at firrtl.antlr.FIRRTLParser.type(FIRRTLParser.java:600)
    at firrtl.antlr.FIRRTLParser.field(FIRRTLParser.java:740)
    at firrtl.antlr.FIRRTLParser.type(FIRRTLParser.java:647)
    at firrtl.antlr.FIRRTLParser.field(FIRRTLParser.java:740)
    at firrtl.antlr.FIRRTLParser.type(FIRRTLParser.java:647)
    at firrtl.antlr.FIRRTLParser.field(FIRRTLParser.java:740)
    at firrtl.antlr.FIRRTLParser.type(FIRRTLParser.java:647)
    at firrtl.antlr.FIRRTLParser.port(FIRRTLParser.java:418)
    at firrtl.antlr.FIRRTLParser.module(FIRRTLParser.java:287)
    at firrtl.antlr.FIRRTLParser.circuit(FIRRTLParser.java:189)
    at firrtl.Parser$.$anonfun$parseCharStream$1(Parser.scala:42)
    at firrtl.Parser$$$Lambda$94/1642030774.apply(Unknown Source)
    at firrtl.Utils$.time(Utils.scala:186)
    at firrtl.Parser$.parseCharStream(Parser.scala:33)
    at firrtl.Parser$.parseFile(Parser.scala:25)
    at firrtl.Driver$.$anonfun$getCircuit$5(Driver.scala:200)
    at firrtl.Driver$$$Lambda$93/1571967156.apply(Unknown Source)
    at scala.Option.getOrElse(Option.scala:121)
    at firrtl.Driver$.$anonfun$getCircuit$3(Driver.scala:183)
    at firrtl.Driver$$$Lambda$91/802600647.apply(Unknown Source)
    at scala.Option.getOrElse(Option.scala:121)
    at firrtl.Driver$.$anonfun$getCircuit$1(Driver.scala:183)
    at firrtl.Driver$$$Lambda$88/2041416495.apply(Unknown Source)
chisel out-of-memory jvm
1 Answers

This is a known limitation of the parser and there are two workarounds, as suggested by Jack in this comment:

  1. Use protobuf as the interface between Chisel and FIRRTL. Instead of using chisel3.Driver.dumpFirrtl, use chisel3.Driver.dumpProto. The FIRRTL command line utility will automatically infer .pb files as protobuf and use the appropriate deserializer as opposed to the ANTLR parser.

  2. Use chisel3.Driver to call FIRRTL directly instead of using the FIRRTL command line utility. This will still cause FIRRTL to be emitted for you to examine, but internally the Chisel representation is converted directly to FIRRTL without going through parsing. Try:

val args = Array("-o", "mnist_cnn_v4.v",
                 "-X", "verilog",
                 "--no-check-comb-loops",
                 "--no-dce",
                 "--info-mode=ignore",
                 "-ll Trace")

chisel3.Driver.execute(args, () => new Foo) /* change Foo to your top module */

The latter in-memory conversion was added in Chisel3#829 and protobuf support was added in FIRRTL#832. Jack has some experiments referenced there of reading a 420MB FIRRTL file using different methods. Summarily, try to avoid using the parser if you have big FIRRTL files.

user_38705490
Related