Lazily extract lines from large file

Viewed 2801

I'm trying to grab 5 lines by their line numbers from a large (> 1GB) file with Clojure. I'm almost there but am seeing some strange things, and I want to understand what's going on.

So far I've got:

(defn multi-nth [values indices]
  (map (partial nth values) indices))

(defn read-lines [file indices]
  (with-open [rdr (clojure.java.io/reader file)]
    (let [lines (line-seq rdr)]
      (multi-nth lines indices))))

Now, (read-lines "my-file" [0]) works without a problem. However, passing in [0 1] gives me the following stacktrace:

java.lang.RuntimeException: java.io.IOException: Stream closed
        Util.java:165 clojure.lang.Util.runtimeException
      LazySeq.java:51 clojure.lang.LazySeq.sval
      LazySeq.java:60 clojure.lang.LazySeq.seq
         Cons.java:39 clojure.lang.Cons.next
          RT.java:769 clojure.lang.RT.nthFrom
          RT.java:742 clojure.lang.RT.nth
         core.clj:832 clojure.core/nth
         AFn.java:163 clojure.lang.AFn.applyToHelper
         AFn.java:151 clojure.lang.AFn.applyTo
         core.clj:602 clojure.core/apply
        core.clj:2341 clojure.core/partial[fn]
      RestFn.java:408 clojure.lang.RestFn.invoke
        core.clj:2430 clojure.core/map[fn]

It seems that the stream is being closed before I can read the second line from the file. Interestingly, if I manually pull out a line from the file with something like (nth lines 200), the multi-nth call works for all values <= 200.

Any idea what's going on?

2 Answers
Related