Lets say I have 2 different tables that needed to be filled. Each table can have different number of source files. I want to fill the tables in parallel but each source file will be read consequently. example: Table A: 3 source files, Table B: 1 source files
tables = {'A': [sourcefileA_1, sourcefileA_2,sourcefileA_3], 'B':[sourcefileB_1]}
threads = []
tables.each do |table,source_files|
threads << Thread.new { do_something(table, source_files)}
end
threads.each(&:join)
def do_something(table, source_files)
source_files.each do |s_file|
#do something
end
So, the number of threads depends on the number of tables I have. Lets say now I have 500.000 tables and just a normal Mac Pro. I thought of using ThreadPool, however, if some error occurs in the thread it is not shown. In comparison if I use just threads the error is still shown.Is there any gem that can help me to use thread pool and that will throw an error if it occurs in one of the threads.
Regarding not showing the error, consider following:
require 'thread/pool' # from the ruby-thread gem
pool = Thread.pool(4)
pool.process do
puts(b)
end
b is not specified but it does not show me the normal error message.