How to process big JSON array with Apache Camel

user_4256563 0
Viewed 1458

I want to achieve something similar to CSV streaming processing:

//read input file
.split(body().tokenize("\n", 100, false)).streaming()
            .unmarshal(new BindyCsvDataFormat( ...

I can control the content of input file, for example, I can have each JSON object on new line without JSON start array and comma after each object:

{"id": "foo1"}
{"id": "foo2"}
...

And then follow the same flow as in CSV (split and stream), but I can't unmarshal using ListJacksonDataFormat or .json(JsonLibrary.Jackson)

How to do this? Or is there another way of reading big JSON array?

Note: This processing has to be fast, so I can't afford unmarshal to csv then marshal JSON as explained here (which seems a gross workaround).

jackson apache-camel json java
2 Answers
user_4064290

I ended up with the following solution:

.split(body().tokenize("\n", 1_000, false))
.streaming()
.process(exchange -> {
    String[] body = exchange.getIn().getBody(String.class).split("\n");
    var records = new ArrayList<FooBar>(body.length);
    for(String line: body) {
         records.add(objectMapper.readValue(line, FooBar.class));
    }
    exchange.getIn().setBody(records);
})

objectMapper is com.fasterxml.jackson.databind.ObjectMapper

For example, a 3.5 GB file was processed in ~1.2 min.

user_42565630
Related