Ruby CSV.foreach start at specific row

user_1798531 0
Viewed 4039

I've seen a couple posts for this with no real answers or out-of-date answers, so I'm wondering if there are any new solutions. I have an enormous CSV I need to read in. I can't call open() on it bc it kills my server. I have no choice but to use .foreach().

Doing it this way, my script will take 6 days to run. I want to see if I can cut that down by using Threads and splitting the task in two or four. So one thread reads lines 1-n and one thread simultaneously will read lines n+1-end.

So I need to be able to only read in the last half of the file in one thread (and later if I split it into more threads, just a specific line through a specific line).

Is there anyway in Ruby to do this? Can this start at a certain row?

CSV.foreach(FULL_FACT_SHEET_CSV_PATH) do |trial|

EDIT: Just to give an idea of what one of my threads looks like:

threads << Thread.new { 
CSV.open('matches_thread3.csv', 'wb') do |output_csv|

  output_csv << HEADER
  count = 1
  index = 0

    CSV.foreach(CSV_PATH) do |trial|
        index += 1
        if index > 120000 
            break if index > 180000
            #do stuff
        end
    end
end
}

But as you can see, it has to iterate the file until it gets to record 120,000 before it starts. So the goal would be to eliminate reading all of the rows before row 120,000 by starting to read at row 120,000.

csv multithreading ruby-on-rails foreach ruby
3 Answers

If still relevant, you can do something like this using .with_index after :

rows_array = []

CSV.foreach(path).with_index do |row, i|
  next if i == 0 #skip first row
  rows_array << columns.map { |n| row[n] }
end
user_108426170

But as you can see, it has to iterate the file until it gets to record 120,000 before it starts. So the goal would be to eliminate reading all of the rows before row 120,000 by starting to read at row 120,000.

Impossible. Content of a CSV file is just a blob of text, with some commas and newlines. You can't know at which offset in the file row N starts without knowing where row N-1 ends. And to know this, you have to know where row N-1 starts (see recursion?) and read the file until you see where it ends (encounter a newline that is not part of field value).

Exception to this is if all your rows are of fixed size. In which case, you can seek directly to offset 120_000 * row_size. I am yet to see a file like this, though.

user_1258160

As per my understanding towards your Question in Ruby way it may help you.

require 'csv'
csv_file = "matches_thread3.csv"
# define one Constant Chunk Size for Jobs
CHUNK_SIZE = 120000
# split - by splitting (\n) will generate an array of CSV records
# each_slice - will create array of records of CHUNK_SIZE defined

File.read(csv_file).split("\n").drop(1).each_slice(CHUNK_SIZE).with_index 
do |chunk, index|   

  data = []
   # chunk will be work as multiple Jobs of 120000 records 

  chunk.each do |row|
   data << r
   ##do stuff
  end
end
user_117236060
Related