How to truncate csv file to n of rows not reading the whole file

Viewed 75

I have big csv(12 gb), so I can't read it in memory, and I need only 100 rows of them and save it back(truncate). Has java such api?

4 Answers

The other answers create a new file from the original file. As I understand it, you want to truncate the original file instead. You can do that quite easily using RandomAccessFile:

    try (RandomAccessFile file = new RandomAccessFile(FILE, "rw")) {
        for (int i = 0;  i < N && file.readLine() != null; i++)
            ;  // just keep reading
        file.setLength(file.getFilePointer());
    }

The caveat is that this will truncate after N lines, which is not necessarily the same thing as N rows, because CSV files can have rows that span multiple lines. For example, here is one CSV record that has a name, address, and phone number, and spans multiple lines:

Joe Bloggs, "1 Acacia Avenue,
Naboo Town,
Naboo", 01-234 56789

If you are sure all your rows only span one line, then the above code will work. But if there is any possibility that your CSV rows may span multiple lines, then you should first parse the file with a suitable CSV reader to find out how many lines you need to retain before you truncate the file. OpenCSV makes this quite easy:

    final long numLines;
    try (CSVReader csvReader = new CSVReader(new FileReader(FILE))) {
        csvReader.skip(N);  // Skips N rows, not lines
        numLines = csvReader.getLinesRead();  // Gives number of lines, not rows
    }

    try (RandomAccessFile file = new RandomAccessFile(FILE, "rw")) {
        for (int i = 0; i < numLines && file.readLine() != null; i++)
            ;  // just keep reading
        file.setLength(file.getFilePointer());
    }

You should stream a file : read it line by line

For example :

    CSVReader reader = new CSVReader(new FileReader("myfile.csv"));
    String [] nextLine;

    // the readnext => Reads the next line from the buffer and converts to a string array.

    while ((nextLine = reader.readNext()) != null) {
        System.out.println(nextLine); 
    }

If you need just a hundred lines, reading just that small portion of the file into memory would be really quick and cheap. You could use the Standard Library file APIs to achieve this quite easily:

val firstHundredLines = File("test.csv").useLines { lines ->
  lines.take(100).joinToString(separator = System.lineSeparator())
}
File("test.csv").writeText(firstHundredLines)

Possible solution

File file = new File(fileName);

// collect first N lines
String newContent = null;
try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
    newContent = reader.lines().limit(N).collect(Collectors.joining(System.lineSeparator()));
}

// replace original file with collected content
Files.write(file.toPath(), newContent.getBytes(), StandardOpenOption.TRUNCATE_EXISTING);
Related