I'm writing the front-end part of an interpreter and I initially disliked the idea of just dumping all the source files into memory and then referencing that text directly. So the tokenizer reads from a char buffers and builds the token stream.
However, I have reached the parsing side of things and it hit me that because I would want to output nice errors and warnings that show the malformed line of source code. I guess I could put column numbers in the tokens, but then by error messages would be like getting directions by telephone: "It's in file X, on line Y, column Z, right next to the curly brace, you know the one. If you hit the semicolon, you've gone to far."
I seem to have put myself into a situation where I want to have my cake and eat it too. I want nice messages, but I don't want to hog memory.
It there something I'm missing? Or is loading the source in memory the way to go?