(import java.nio.file.Files)
(import java.nio.file.Paths)
(import java.util.stream.Stream)
(def path
(Paths/get "."
(into-array ["data" "10000000.test.log"])))
(def stream
(Files/lines path))
This way I have:
stream
#object[java.util.stream.ReferencePipeline$Head 0x50129b8f
"java.util.stream.ReferencePipeline$Head@50129b8f"]
Is there a way to iterate over this without running out of memory? The suggestion on SO are not really helpful. The files is ~1G.
Do the same thing you would do in java. Call interop methods on it, like (.map stream ...), or whatever else, and collect it to a list at the end, or reduce it, or whatever you need to do.
Clojure functions don't implement the interfaces that a stream expects, such as java.util.function.Function, so you'll need to use reify instead of a simple fn:
(.map stream (reify java.util.function.Function (apply [this x] (foo x))))
I wanted to iterate over the stream Java / imperative style to avoid blowing up the heap. I do not need to reduce on the stream, I need to process every line and take out one field and send it out. I think for this I probably better off with doseq.
(doseq
[l (iterator-seq (.iterator stream))]
(println l))