Compiler test cases or how to test a compiler

user_95321 0
Viewed 18250

Compilers like all software, would also be prone to bugs, logical errors.

How does one validate the output generated by the compiler. Typically, my question is(are)

  • How to validate that the machine code generated is correct?

  • How to ensure that the machine code generated is according to the language specification.

  • Does it make sense to just pick an open source project (in C if one is also writing a compiler in C) to just compile it through the "compiler". In that case also, how do judge that the compiler is behaving as expected.

  • Are there any formal test cases (literature) provided by the language standards committee that a "language complying" compiler has to satisfy?

  • What are the sure "give aways" that the problem in a program compiled by a compiler is a compiler bug and not a program bug.

    - Any examples where mainstream compilers get confused and compile the code wrong?

Links to any literature would be appreciated.

compiler-construction
7 Answers

Good test suites for real languages are expensive to create and maintain. There's a reason that the Plum Hall test suite, which is industry standard for ANSI C, is so bloody expensive.

George Necula's translation validation is a brilliant idea but also quite expensive to implement.

The one thing that's cheap and easy is this: maintain a suite of regression tests, and every time you fix a bug in your compiler, put a suitable test into your regression suites. With compilers, it's unbelievable how easy it is to keep reintroducing the same bug over and over. Disciplined additions to your regression suite will prevent that, and they don't cost much.

Norman_Ramsey_416610

There are several compiler test suites out there. We've had some luck using the Plum Hall test suite for a C compiler. It consists of a large set of C code specifically written to test against the language standard. It verifies that the compiler can handle the language syntax and semantics.

Trent_90830

The general practice is to create a large set of small programs that each demonstrate one aspects of the compiler. These will include both program that compile and ones that shouldn't. General the ASM coming out the back end is not checked but rather the program is run and it's output checked. As for how to make sure there are no bugs in the test cases: make them small, as in 5-10 lines each.

These test suites can be very large as in hundreds to thousands of tests (for example: an out of date test suite for the D programming language) and generally include one or more test cases for every bug ever reported.

BCS_13430

The Eiffel compiler is open source and has an extensive library of test cases and internal design contracts.

http://dev.eiffel.com

user_459380
plinth_204810
Related