Handling files bigger than 2GB while downloading

Vishal Sain 0
Viewed 86

I have been trying to download a file using MediaHttpDownloader which is greater than 2gb. After downloading it for a while i am getting the out of memory error. Which i think is caused by the mediaHttpDownloader trying to write it to Outputstream as output stream can't handle data so big.

Code:

MediaHttpDownloader mediaHttpDownloader = new MediaHttpDownloader(HTTP_TRANSPORT, new HttpCredentialsAdapter(credentials));
mediaHttpDownloader.download(requestUrl, outputStream);

Error :

Caused by: java.lang.OutOfMemoryError: null
    at java.base/java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:125)
    at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:119)
    at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
    at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)

Would love some help/suggestions on the best ways to handle download of a very large file.

Thanks

outputstream java
1 Answers

Every 500mb acquired call System.gc() , get the free memory from Runtime class. You just need to put some code in to check. Also, for huge data you need to setup parameters for using the G1 garbage collector on your program command line. https://www.oracle.com/technetwork/tutorials/tutorials-1876574.html Note, With URLConnection and HttpURLConnection you can setup that system for setDoInput and use streams. You should also have a memory share of RAM scheme on the command line for heap and stack E.t.c. Check whether running in X32 or X64 because x32 has limited finite assignable heap for RAM stack assign is max 1mb.

Samuel Marchant0
Related