Gson library is not working reliably as intended for parsing large JSON file

user_9639295 0
Viewed 247

I have to parse a really huge JSON file (file size can get to several GBs), so I cannot just load the entire JSON String into the memory and parse it into an object, I have to somehow read the JSON String line by line and parse it. I am currently using JsonReader from Gson library, which was working great, but recently I discovered that it occasionally throws an error saying Unterminated string at line 1 column xxxxxxxxx path $.fieldname[random index].fieldname[random index].fieldname, but when I parsed it using a different library like Jackson, the parsing went flawlessly (this file is not that huge, only 50 MB so I can just load it into the memory and parse it into an object), so is this a BUG in Gson? And if it is, is there any other JAVA library I can use to do the same thing? I will be appreciated for any answer!

PS: I am using gson-2.8.2

EDIT: I have tested the same file again using Gson, the same error occurred but at the different line and different position, so is it confirmed this is BUG in Gson?

gson json java
2 Answers

Looks like you should check the GitHub issues for Gson: https://github.com/google/gson/issues Apart from that, a minimum example to reproduce that would be good; you could even generate such a file to make the example self-contained ;-)

Btw, please change the heading as you apparently know how to do that. It just does not work reliably with Gson...

user_140477240

I tested the parsing with Jackson library as well and still got the same type of error, however, throughout many tests, it turns out Gson and Jackson libraries can have a problem (not always) handling JSON file that is NOT pretty printed (meaning that the JSON does not have proper indentation), because all the JSON files I tested are putting the entire JSON string on a single line (technically still a legit JSON), after I formatted it to have indentations, the parsing went through successfully (both Gson and Jackson), hope this can help anyone who encountered the same issue as I did

user_96392950
Related