how to avoid use of toString().getBytes("UTF-8") to avoid OOM error? Is there any better approach to convert to byte[] from StringWriter?

user_8086483 0
Viewed 1172

I have a reporting web application which generates reports. The application gets data from a database and stores data into a StringWriter object. I have to get this data in a byte array format to create a csv file and send it to browser.

Below is the code snippet

 return new FileTransfer(fileName, reportType.getMimeType(),
                    new ByteArrayInputStream(generateCSV(reportType, grid, new DataList(), params).toString().getBytes("UTF-8")));

where generateCSV returns a StringWriter object, then to convert it into byte array I am calling toString and then getBytes() method. Below is what the generateCSV method looks like

StringWriter generateCSV(ReportType reportType, GridConfig grid, DataList dataList, String params) {......}

The problem is that when my report has huge records (more than 1 million), the getBytes() method fails with

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

The whole report data when converted to String object has a huge number of characters (billions of it). The .getBytes("UTF-8") method convert it into array, each array element as one character. And for 1 million records, the character are exceeding the MAX JVM ARRAY size limit (https://plumbr.io/outofmemoryerror/requested-array-size-exceeds-vm-limit).

Now how can I avoid use of toString().getBytes("UTF-8") to avoid OOM error? Is there any better approach to convert to byte array from StringWriter?

out-of-memory jvm arrays java
2 Answers

The StringWriter holds its content in the memory. So it's not a good approach to use it with large files.

You should try to chunk the File directly to the InputStream without the StringWriter in the middle. What about your own InputStream implementation which reads and convert the file to csv on the fly.

Can you show us the generateCSV method?

user_104070750

It’s strange to receive the result of generateCSV as a StringWriter; the preferred solution would be to let the method write to a target while generating, so you don’t have the entire contents in memory.

In either case, you should resort to the FileTransfer(String, String mimeType, OutputStreamLoader) constructor, to receive the target OutputStream when it is time to write the actual data.

When you can’t avoid the intermediate StringWriter, you should at least avoid calling toString on it, as constructing a String implies creating a copy of the entire buffer.

So a solution could look like:

return new FileTransfer(fileName, reportType.getMimeType(), new OutputStreamLoader() {
    public void close() {}
    public void load(OutputStream out) throws IOException {
        // the best would be to let generateCSV write to out directly
        // otherwise use:
        StringBuffer sb = generateCSV(reportType, grid, new DataList(), params).getBuffer();
        Writer w = new OutputStreamWriter(out, "UTF-8")
        final int bufSize = 8192;
        for(int s = 0, e; s < sb.length(); s = e) {
            e = Math.min(sb.length(), s + bufSize);
            w.write(sb.substring(s, e));
        }
        w.flush(); // let the caller close the OutputStream
    }
});

An alternative to StringWriter would be CharArrayWriter, which has a writeTo​(Writer out), which eliminates the need to implement a manual copying loop and might be even more efficient. But, as said, refactoring generateCSV to write directly to a target would be even better.

user_27114880
Related