I'm currently working on kind of compiler that was built making use of sablecc.
Long story short, the compiler will take as input both specification files(this is what we're parsing) and .class files and will instrument the .class files bytecode so to make sure that when running the .class files, any of the specifications is not being violated (this is a bit like jml/code contracts! but way more powerful).
We have some dozens of system tests that cover a large part of the analysis phase (related with making sure the specifications make sense, and that they also are in concordance with the .class files they are supposed to specify).
We divided them in two sets: the valid tests and the invalid tests.
The valid tests are comprised of source code files that when compiled by our compiler should should pop up no compiler errors / warnings.
The invalid tests are comprised of source code files that when compiled by our compiler should should pop up at least one compiler error / warning.
This has served us well while we were in the analysis phase. The question now is on how to test the code generation phase. I've done, in the past, system tests over a little compiler I've developed on a compilers course. Each test would consist of a couple of source files of that language and a output.txt. When running the test, I'd compile the source files and then run its main method, checking that the output result would be equal to output.txt. All of this was automated, of course.
Now, dealing with this bigger compiler/bytecode-instrumentator, things are not so easy. It's no easy task to replicate what I've done with my simple compiler. I guess the way to go is to lean back from system tests at this stage, and focus on unit-tests.
As any compiler developer knows, a compiler consists of lots of visitors. I am not too sure on how to proceed with unit-testing them. From what I've seen, most of the visitors are calling a counterpart class that has methods related with that visitor (I guess the idea was to keep the SRP for the visitors).
There are a couple of techniques I can take to unit-test my compiler:
Unit testing each one of the visitor's methods separately. This seems to be a good idea for a stackless visitor, but looks like a terrible idea for visitors that use one (or more) stacks. I then go about also unit-testing each of the other methods from standard(read, non-visitors) classes the traditional way.
Unit testing the whole visitor in one go. That is, I create tree that I then visit. In the end, I verify if the symbol table was correctly updated or not. I do not care about mocking its dependencies.
The same as 2), but now mocking the visitor's dependencies.
What others?
I still have the problem that the unit-tests will be very tightly coupled with sabbleCC's AST (which tbh is really ugly).
We are currently not making any new tests, but I'd like to bring the train back on track, as I am sure that not testing the system is the same as feeding a monster that sooner or later will come back to bite us in the butt when we least expect it ;-(
Has anyone had any experience with compiler testing that could give some awwweeeesome advice on how to proceed now? I'm kinda lost here !