Multi threading for reading CSV Files in Julia

user_14847557 0
Viewed 644

Is there a way to implement multi-threading on CSV.read in Julia? A good example to read big files has been presented in Parallelism for reading a large file in Julia. But since I have to frequently change my datasets these approaches may not be applicable.

using CSV
file = ("C:\\Users\\User\\Desktop\\Datasets\\X_train_sat4.csv")
@time df = CSV.read(file, DataFrame)

Output:
69.469112 seconds (6.29 M allocations: 9.767 GiB, 0.76% gc time)
29723 rows × 2456 columns

I have used the steps recommended here Speed up loading and compilation time but these improves only the first loading time.

Thanks in advance!

julia csv multithreading
1 Answers

CSV.jl supports multi-threading. You can select how many threads you want to use with tasks keyword argument. You cannot use more than number of threads that your Julia process was started with.

Here is a sample timing on reading a file having 10^8 rows and 10 columns:

julia> @time CSV.read("test.csv", DataFrame, tasks=1); # one thread
 75.190387 seconds (23.24 M allocations: 9.253 GiB, 1.24% gc time)

julia> @time CSV.read("test.csv", DataFrame, tasks=2); # two threads
 43.078513 seconds (4.34 M allocations: 8.044 GiB, 2.30% gc time)

By default number of tasks used is set to Threads.nthreads().

user_12695670
Related