A legacy application I work with has a funky data format called SGS. I've considered and started on a number of brute-force solutions including a hand-generated finite state machine and a custom recursive descent parser, but I'm striving to build an application where the amount of (non-library) source code is just enough to express what needs to be done.
So I've been looking at Clojure-based parsers. I've fiddled with
- Clearley,
- Instaparse,
- Parse-EZ, and
- Squarepeg.
None of these have enough documentation/support on the 'net to get me off to a running start. So I'm looking for someone with experience with one of these tools (or a good alternative) to give me a hand.
Here's the data language:
Data is represented by rows having a label (starting in col. 1) and 1 or more fields, separated by one or more spaces.
Fields consist of one or more subfields, separated by commas. Commas may be followed by spaces for legibility but these aren't significant.
Labels are identifiers consisting of characters in the set [-$0-9A-Z_*%] and need not be unique.
Subfields are either identifiers or quoted strings or missing (nil). Quoted strings are delimited by 2 leading single quotes and 2 trailing single quotes. Quoted strings don't contain single quotes, so there's no quote nesting or escaping to worry about.
Space-dot-space starts a rest-of-line comment. Space-dot at the end of a line is an empty rest-of-line comment. Dot-space at the beginning of a line makes the entire line a comment. A line consisting of only a dot is also a line comment.
Rows may be continued across two or more lines. A semicolon as the last non-blank, non-comment character on a line means the row is continued on the next line as if both the semicolon and line break were nonexistent.
Semicolons and dots (with or without spaces) have no special significance inside comments or quoted strings.
Example:
. Comment
.
LAB1 F1S1 . Minimal data row, with line comment
LAB1 F1S1,F1S2,F1S3 F2S1 F3S1 . 2nd row with same label
LAB2 , , , F1S4 ''Field #2 (only 1 subfield)'' F3S1,,F3S3
LAB99 F1S1, . Field 1 has 2 subfields, 2nd is nil
LAB3 F1S1,F1S2, ;
F1S3 ;
F2S1 . Row continued over 3 lines.
Hand-parsing my example, I would like a result like this:
[
("LAB1" ["F1S1"])
("LAB1" ["F1S1" "F1S2" "F1S3"] ["F2S1"] ["F3S1"])
("LAB2" [nil nil nil "F1S4"] ["Field #2 (only 1 subfield"] ["F3S1" nil "F3S3"])
("LAB99" ["F1S1" nil])
("LAB3" ["F1S1" "F1S2" "F1S3"] ["F2S1"])
]
UPDATE:
@edwood suggested showing an implementation of my own for people to use as a starting point. I had hesitated to do that to avoid pre-biasing people in a particular direction, but with the dearth of responses this may be "better than nothing."
Here, then, is my own InstaParse solution, which sorta-works:
SGS = (<COMMENT_LINE> / DATA_LINES) *
COMMENT_LINE = #' *\\.(?: [^\\n]*)?\\n'
DATA_LINES = LABEL FIELDS SEPARATOR? (LINE_COMMENT | '\\n')
LABEL = IDENTIFIER
FIELDS = '' | (SEPARATOR FIELD)+
SEPARATOR = CONTINUATION #' +' | #' +' (CONTINUATION #' *')?
CONTINUATION = #'; *\\n'
LINE_COMMENT = #' .[^\\n]*\\n'
FIELD = SUBFIELD (',' SEPARATOR? SUBFIELD)*
SUBFIELD = IDENTIFIER | QUOTED_STRING | ''
IDENTIFIER = #'[-$0-9A-Z_*%]+'
QUOTED_STRING = #'\\'\\'[^\\']*\\'\\''
While debugging, it managed to process 249 lines before hitting an error I needed to debug. But once I fixed that, and it presumably went to work on my whole 431 lines of data, and ended up choking, after about 2 minutes, with
CompilerException java.lang.OutOfMemoryError: Java heap space, compiling:(sgs2.clj:40:13)
I moved the easily regexp-handled stuff to regexps, and this seems to have helped performance. Comment lines, for example, are now trivial to parse because they directly match a single regexp, or not.
If I chop my input data down to 228 lines, the parser runs and produces a correct result in 16 seconds. I think this is a very long time for a very small amount of data. Am I doing something dramatically wrong?