Fast way to write millions of small text files in Java?

Viewed 5910

I have to dump 6 million files which contain around 100-200 characters, and it's painfully slow. The actual slow part is the file writing, if I comment that part out (the call to the WriteSoveraFile method) the whole thing runs in 5-10 minutes. As it is, I ran it overnight (16 hours) and got done with 2 million records.

  1. is there any faster method?

  2. Would I be better off creating an array of arrays and then dumping it all at once? (my system only has 4 GB, wouldn't it die from the 6 GB of data consumed by this?)

Here is the procedure:

public static void WriteSoveraFile(String fileName, String path, String contents) throws IOException {

    BufferedWriter bw = null;

    try {
        String outputFolderPath = cloGetAsFile( GenCCD.o_OutER7Folder ).getAbsolutePath() ;
        File folder = new File( String.format("%1$s/Sovera/%2$s/", outputFolderPath, path) );  

        if (! folder.exists()) {
            folder.mkdirs();

/*          if (this.rcmdWriter != null)
              this.rcmdWriter.close();
*/        
        } 

        File file = new File( String.format("%1$s/%2$s", folder.getAbsolutePath(),fileName) );

        // if file doesnt exists, then create it
        if (!file.exists()) {
            file.createNewFile();
            FileWriter fw = new FileWriter(file.getAbsoluteFile());
            bw = new BufferedWriter(fw);
            bw.write(contents);
            bw.close();
        }
/*      else {
            file.delete();  // want to delete the file??  or just overwrite it??
            file.createNewFile();*/

    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            if (bw != null) bw.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }
}
3 Answers
Related