Search in large CSV as fast as Guava Splitter

Viewed 351

Since Java 8 was released I found out I don't need over 2 MB Google Guava in my projects since I can replace most of it with plain Java. However I really liked nice Splitter API which was both quite fast at the same time. And what is most important - did splitting lazily. It seems to be replaceable with Pattern.splitAsStream. So I prepared quick test - finding a value in the middle of long string (i.e. splitting the whole string does not make sense).

package splitstream;


import com.google.common.base.Splitter;
import org.junit.Assert;
import org.junit.Test;

import java.util.StringTokenizer;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

public class SplitStreamPerfTest {

    private static final int TIMES = 1000;
    private static final String FIND = "10000";

    @Test
    public void go() throws Exception {
        final String longString = IntStream.rangeClosed(1,20000).boxed()
                .map(Object::toString)
                .collect(Collectors.joining(" ,"));

        IntStream.rangeClosed(1,3).forEach((i) -> {
            measureTime("Test " + i + " with regex", () -> doWithRegex(longString));
            measureTime("Test " + i + " with string tokenizer", () -> doWithStringTokenizer(longString));
            measureTime("Test " + i + " with guava", () -> doWithGuava(longString));
        });

    }

    private void measureTime(String name, Runnable r) {
        long s = System.currentTimeMillis();
        r.run();
        long elapsed = System.currentTimeMillis() - s;
        System.out.println("Check " + name +" took " + elapsed + " ms");
    }

    private void doWithStringTokenizer(String longString) {

        String f = null;
        for (int i = 0; i < TIMES; i++) {
            StringTokenizer st = new StringTokenizer(longString,",",false);
            while (st.hasMoreTokens()) {
                String t = st.nextToken().trim();
                if (FIND.equals(t)) {
                    f = t;
                    break;
                }
            }
        }
        Assert.assertEquals(FIND, f);
    }


    private void doWithRegex(String longString) {
        final Pattern pattern = Pattern.compile(",");
        String f = null;
        for (int i = 0; i < TIMES; i++) {
            f = pattern.splitAsStream(longString)
                    .map(String::trim)
                    .filter(FIND::equals)
                    .findFirst().orElse("");
        }
        Assert.assertEquals(FIND, f);
    }


    private void doWithGuava(String longString) {
        final Splitter splitter = Splitter.on(',').trimResults();
        String f = null;
        for (int i = 0; i < TIMES; i++) {
            Iterable<String> iterable = splitter.split(longString);
            for (String s : iterable) {
                if (FIND.equals(s)) {
                    f = s;
                    break;
                }
            }
        }
        Assert.assertEquals(FIND, f);
    }
}

The results are (after a warm-up)

Check Test 3 with regex took 1359 ms
Check Test 3 with string tokenizer took 750 ms
Check Test 3 with guava took 594 ms

How to make the Java implementation as fast as Guava? Maybe I'm doing it wrong?

Or maybe you know any tool/library as fast as Guava Splitter that does not involve pulling tons of unused classes just for this one?

4 Answers
Related