How to copy large data files line by line?

user_1194415 0
Viewed 230

I have a 35GB CSV file. I want to read each line, and write the line out to a new CSV if it matches a condition.

try (BufferedWriter writer = Files.newBufferedWriter(Paths.get("source.csv"))) {
    try (BufferedReader br = Files.newBufferedReader(Paths.get("target.csv"))) {
        br.lines().parallel()
            .filter(line -> StringUtils.isNotBlank(line)) //bit more complex in real world
            .forEach(line -> {
                writer.write(line + "\n");
        });
    }
}

This takes approx. 7 minutes. Is it possible to speed up that process even more?

java-stream java-io java
3 Answers

If it is an option you could use GZipInputStream/GZipOutputStream to minimize disk I/O.

Files.newBufferedReader/Writer use a default buffer size, 8 KB I believe. You might try a larger buffer.

Converting to String, Unicode, slows down to (and uses twice the memory). The used UTF-8 is not as simple as StandardCharsets.ISO_8859_1.

Best would be if you can work with bytes for the most part and only for specific CSV fields convert them to String.

A memory mapped file might be the most appropriate. Parallelism might be used by file ranges, spitting up the file.

try (FileChannel sourceChannel = new RandomAccessFile("source.csv","r").getChannel(); ...
MappedByteBuffer buf = sourceChannel.map(...);

This will become a bit much code, getting lines right on (byte)'\n', but not overly complex.

user_9848230

you can try this:

try (BufferedWriter writer = new BufferedWriter(new FileWriter(targetFile), 1024 * 1024 * 64)) {
  try (BufferedReader br = new BufferedReader(new FileReader(sourceFile), 1024 * 1024 * 64)) {

I think it will save you one or two minutes. the test can be done on my machine in about 4 minutes by specifying the buffer size.

could it be faster? try this:

final char[] cbuf = new char[1024 * 1024 * 128];

try (Writer writer = new FileWriter(targetFile)) {
  try (Reader br = new FileReader(sourceFile)) {
    int cnt = 0;
    while ((cnt = br.read(cbuf)) > 0) {
      // add your code to process/split the buffer into lines.
      writer.write(cbuf, 0, cnt);
    }
  }
}

This should save you three or four minutes.

If that's still not enough. (The reason I guess you ask the question probably is you need to execute the task repeatedly). if you want to get it done in one minutes or even couple of seconds. then you should process the data and save it into db, then process the task by multiple servers.

user_33807390

Thanks to all your suggestions, the fastest I came up with was exchanging the writer with BufferedOutputStream, which gave approx 25% improvement:

   try (BufferedReader reader = Files.newBufferedReader(Paths.get("sample.csv"))) {
        try (BufferedOutputStream writer = new BufferedOutputStream(Files.newOutputStream(Paths.get("target.csv")), 1024 * 16)) {
            reader.lines().parallel()
                    .filter(line -> StringUtils.isNotBlank(line)) //bit more complex in real world
                    .forEach(line -> {
                        writer.write((line + "\n").getBytes());
                    });
        }
    }

Still the BufferedReader performs better than BufferedInputStream in my case.

user_11944150
Related