How to get a random line of a text file in Java?

Viewed 48592

Say there is a file too big to be put to memory. How can I get a random line from it? Thanks.

Update: I want to the probabilities of getting each line to be equal.

7 Answers

Reading the entire file if you want only one line seems a bit excessive. The following should be more efficient:

  1. Use RandomAccessFile to seek to a random byte position in the file.
  2. Seek left and right to the next line terminator. Let L the line between them.
  3. With probability (MIN_LINE_LENGTH / L.length) return L. Otherwise, start over at step 1.

This is a variant of rejection sampling.

Line lengths include the line terminator character(s), hence MIN_LINE_LENGTH >= 1. (All the better if you know a tighter bound on line length).

It is worth noting that the runtime of this algorithm does not depend on file size, only on line length, i.e. it scales much better than reading the entire file.

Here's a solution. Take a look at the choose() method which does the real thing (the main() method repeatedly exercises choose(), to show that the distribution is indeed quite uniform).

The idea is simple: when you read the first line it has a 100% chance of being chosen as the result. When you read the 2nd line it has a 50% chance of replacing the first line as the result. When you read the 3rd line it has a 33% chance of becoming the result. The fourth line has a 25%, and so on....

import java.io.*;
import java.util.*;

public class B {

  public static void main(String[] args) throws FileNotFoundException {
     Map<String,Integer> map = new HashMap<String,Integer>();
     for(int i = 0; i < 1000; ++i)
     {
        String s = choose(new File("g:/temp/a.txt"));
        if(!map.containsKey(s))
           map.put(s, 0);
        map.put(s, map.get(s) + 1);
     }

     System.out.println(map);
  }

  public static String choose(File f) throws FileNotFoundException
  {
     String result = null;
     Random rand = new Random();
     int n = 0;
     for(Scanner sc = new Scanner(f); sc.hasNext(); )
     {
        ++n;
        String line = sc.nextLine();
        if(rand.nextInt(n) == 0)
           result = line;         
     }

     return result;      
  }
}

Either you

  1. read the file twice - once to count the number of lines, the second time to extract a random line, or

  2. use reservoir sampling

Use RandomAccessFile:

  1. Construct a RandomAccessFile, file
  2. Get the length of that file, filelen, by calling file.length()
  3. Generate a random number, pos, between 0 and filelen
  4. Call file.seek(pos) to seek to the random position
  5. Call file.readLine() to get to the end of the current line
  6. Read the next line by calling file.readLine() again

Using this method, I've been sampling lines from the Brown Corpus at random, and can easily retrieve a 1000 random samples from randomly chosen files in a few seconds. If I tried to do the same by reading through each file line-by-line it would take me much longer.

The same principle can be used for selecting random elements from a list. Rather than reading through the list and stopping at a random place, if you generate a random number between 0 and the length of the list, then you can index directly into the list.

Reading a random line from a file in java:

public String getRandomLineFromTheFile(String filePathWithFileName) throws Exception {

        File file = new File(filePathWithFileName); 
        final RandomAccessFile f = new RandomAccessFile(file, "r");
        final long randomLocation = (long) (Math.random() * f.length());
        f.seek(randomLocation);
        f.readLine();
        String randomLine = f.readLine();
        f.close();
        return randomLine;
    }

Use a BufferedReader and read line wise. Use the java.util.Random object to stop randomly ;)

Related