Efficient iteration of large text file

user_4247945 0
Viewed 108

So I have the following:

val bufferedSource = io.Source.fromFile("""C:\Users\something\workspace\""" + fileName)
val lines = bufferedSource.getLines

I would like to select at random, a start index and an end index and iterate through lines in this range while printing to a new file. Is there a way to access the elements in lines iterator by index?

My first attempt was to copy over the data to a ListBuffer:

var lineArr = ListBuffer[String]()
for (line <- lines) {
    lineArr += line
}

Therafter if I iterate through lineArr in my range, by index, it is really slow.

In what way could I do this efficiently?

Sidenote: If I iterate through lines which contains all elements (which I do not want) it is fast to iterate while writing them to a new file, however I only want a select amount to write.

bigdata iterator scala
3 Answers

So instead of iterating through each line I solved this problem by using slicing. I still create a ListBuffer but I slice it on the start and end index:

lineArrTemp = lineArrTemp.slice(start, end)

And thereafter simply iterate through the ListBuffer iterator, it is efficient.

user_42479450 
  lines
    .drop(startIndex)
    .take(endIndex - startIndex)
    .foreach(writeToFile)
user_42545170
Related