I need to ingest large JSON files whose records may span multiple lines (not files) (depends entirely on how the data provider is writing it).
Elephant-Bird assumes LZO compression, which I know the data provider will not be doing.
The Dzone article http://java.dzone.com/articles/hadoop-practice makes the assumption that the JSON record will be on the same line.
Any ideas, with the exception of squishing the JSON... file will be huge... on how to properly split the file such that the JSON does not break.
Edit: lines, not files