I'm trying to process a huge tar file containing multiple inner XML files to be processed. I have disk constraints, so I have to use an uncompress-in-memory-do-stuff-then-forget approach. I'm using Apache Compress library. So far so good, this is working fine, but I would like now to parallelise the processing of the files to improve the execution time. To do so, we are using reactor.
This looks to be working until, for some reason, data is not been cleaned up after being processed, so it ends up in a java.lang.OutOfMemoryError: Java heap space exception. Other things I have considered is to tune the garbage collector to step in faster, but I'm a bit lost so I appreciate any kind of help.
Getting into the code. This is the Flux.generate I'm using. For every next requested item, it generates an array of bytes containing one uncompressed file from the tar file. Looks like this:
(ArchiveInputStream tarArchive, SynchronousSink<byte[]> sink) -> {
process: try {
ArchiveEntry entry = null;
while ((entry = tarArchive.getNextEntry()) != null) {
if (entry.isDirectory()) continue;
log.info(entry.getName());
var bytes = IOUtils.readRange(tarArchive, (int) entry.getSize());
sink.next(bytes);
break process;
}
log.info("done!!!!");
sink.complete();
} catch (IOException e) {
log.error("Exception getting file entry", e);
sink.error(e);
}
return tarArchive;
}
Later on, the array of bytes is converted into a ByteArrayInputStream to be mapped by JAXB into the internal model:
public Feed processElement(byte[] file) {
var feed = new Feed();
if (file.length != 0) {
try(var is = new ByteArrayInputStream(file)) {
var jaxbUnmarshaller = jaxbContext.createUnmarshaller();
feed = (Feed) jaxbUnmarshaller.unmarshal(is);
} catch (Exception ex) {
log.error("Error during XML unmarshal", ex);
}
}
return feed;
}
And stream is parallelised as follows:
uncompressFiles(tarFileName)
.parallel()
.runOn(parallel())
.map(parser::processElement)
.sequential()
//.....
Let me know if you have any insight either on in this particular case or about how to resolve memory leaks.