I'm hashing (with MD2) a large file parallel using ExecutorService. The process is the following: First, the "inputparser3" method decides if the input is a text or a file, if it is a file, and larger than 100MB, splits the file into chunks and store them in a List of byte[], every chunk is 10MB. Second, the "encrypt3" method hashes every chunk parallel using ExecutorService, for example using 6 threads. After, I use the futures list (with the hashes of the small chunks) to generate a master hash (root hash).
For the example, I download a test file (5GB) from this website: https://testfiledownload.com/ When I tested my code I recognized that it takes forever to finish... about 17 minutes (1000ms), without ExecutorServices and splitting it into chunks it takes only 6 minutes (400ms) on average. I recognized that the threadPool.invokeAll(callableTasks) takes most of the time, and I believe because it has to execute 500 chunks on 6 thread.
My questions are the following:
Is it possible to run 6 chunks at a time, I mean when the first 6 finishes start the second 6...and so on...
Is it possible to improve somehow?
Should I completely remove the parallelization?
My code (in main method):
String largefile = "C:\\Users\\Thend\\Desktop\\test_5gb\\5gb.test";
StopWatch st = new StopWatch();
st.start();
String hash = encrypt3(Objects.requireNonNull(inputparser3(largefile, "FILE")));
st.stop();
System.out.println("Hash: "+hash+" | Execution time: "+String.format("%.5f", st.getTotalTimeMillis() / 1000.0f)+"sec");
The methods:
public static List<byte[]> inputparser3(String PathOrText, String TYPE) throws IOException {
if (TYPE.equals("TXT")) {
List<byte[]> temp = new ArrayList<>();
if(PathOrText.getBytes(StandardCharsets.UTF_8).length > 10485760){ // 10 MB = 10485760 Bytes (in binary)
// Add chunk by chunk
ByteArrayInputStream in = new ByteArrayInputStream(PathOrText.getBytes(StandardCharsets.UTF_8));
byte[] buffer = new byte[1048576]; // 1 MB chunk
int len;
while ((len = in.read(buffer)) > 0) {
temp.add(buffer);
}
return temp;
} else {
// Add whole
temp.add(PathOrText.getBytes(StandardCharsets.UTF_8));
return temp;
}
} else if (TYPE.equals("FILE")) {
List<byte[]> temp = new ArrayList<>();
if(Files.size(Path.of(PathOrText)) > 104857600) { // 100 MB = 104857600 Bytes (in binary)
// Add chunk by chunk
try (FileInputStream fis = new FileInputStream(PathOrText)) {
byte[] buffer = new byte[10485760]; // 10 MB chunk
int len;
while ((len = fis.read(buffer)) > 0) {
temp.add(buffer);
}
return temp;
}
} else {
// Add whole
try (FileInputStream fis = new FileInputStream(PathOrText)) {
byte[] buffer = new byte[(int) PathOrText.length()]; // Add whole
int len;
while ((len = fis.read(buffer)) > 0) {
temp.add(buffer);
}
return temp;
}
}
}
System.err.println("Input type cannot be recognized! Use 'TXT' for text or 'FILE' for file.");
return null;
}
public static String encrypt3(List<byte[]> list) throws ExecutionException, InterruptedException, NoSuchAlgorithmException {
ExecutorService threadPool = Executors.newFixedThreadPool(6);
List<Callable<String>> callableTasks = new ArrayList<>();
MessageDigest md = MessageDigest.getInstance("MD5");
StringBuilder sb = new StringBuilder();
ArrayList<String> temp = new ArrayList<>();
// If there is MORE THAN ONE element in the list
if(list.size() > 1){
list.forEach((n)->{
Callable<String> callableTask = () -> {
sb.setLength(0);
md.reset();
md.update(n);
byte[] hashed_bytes = md.digest();
for (byte hashed_byte : hashed_bytes) {sb.append(Integer.toString((hashed_byte & 0xff) + 0x100, 16).substring(1));}
return sb.toString();
};
callableTasks.add(callableTask);
});
List<Future<String>> futures = threadPool.invokeAll(callableTasks);
threadPool.shutdown();
for (Future<String> future : futures) {
temp.add(future.get());
}
// Create a master hash (root hash)
sb.setLength(0);
md.reset();
md.update(String.join("", temp).getBytes());
byte[] hashed_bytes = md.digest();
for (byte hashed_byte : hashed_bytes) {sb.append(Integer.toString((hashed_byte & 0xff) + 0x100, 16).substring(1));}
return sb.toString();
// If there is ONLY ONE element in the list
} else {
sb.setLength(0);
md.reset();
md.update(list.get(0));
byte[] hashed_bytes = md.digest();
for (byte hashed_byte : hashed_bytes) {sb.append(Integer.toString((hashed_byte & 0xff) + 0x100, 16).substring(1));}
return sb.toString();
}
}
UPDATE:
I have updated my code by removing ExecutorServices, and following the comments of @MarkRotteveel and @g00se.
public static List<byte[]> iparser4(String datatype, String input) throws IOException {
List<byte[]> temp = new ArrayList<>();
if("TXT".equals(datatype)){
ByteArrayInputStream in = new ByteArrayInputStream(input.getBytes(StandardCharsets.UTF_8));
byte[] buffer = new byte[1048576]; // 1 MiB chunk
int len;
while ((len = in.read(buffer)) > 0) {
temp.add(buffer);
}
return temp;
} else if("FILE".equals(datatype)){
try (FileInputStream fis = new FileInputStream(input)) {
byte[] buffer = new byte[1048576]; // 1 MiB chunk
int len;
while ((len = fis.read(buffer)) > 0) {
temp.add(buffer);
}
return temp;
}
}
return null;
}
public static String encrypt4(List<byte[]> list) throws NoSuchAlgorithmException {
MessageDigest md = MessageDigest.getInstance("MD2");
StringBuilder sb = new StringBuilder();
list.forEach((n)-> {
ByteArrayInputStream bai = new ByteArrayInputStream(n);
DigestInputStream dis = new DigestInputStream(new BufferedInputStream(bai), md);
try (bai) {
byte[] buffer = new byte[1048576]; // 1 MiB chunk
while (true) {
int readCount = dis.read(buffer);
if (readCount < 0) {
break;
}
}
byte[] hash = md.digest();
for (byte hashed_byte : hash) {
sb.append(String.format("%02x", hashed_byte));
}
} catch (IOException e) { throw new RuntimeException(e); }
//System.out.println("Hash: "+sb.toString());
});
md.reset();
byte[] masterhash = md.digest(sb.toString().getBytes());
StringBuilder sb2 = new StringBuilder();
for (byte hashed_byte : masterhash) {
sb2.append(String.format("%02x", hashed_byte));
}
return sb2.toString();
}