what's the fastest way to import array data and save the array data with specific format?

user_11941994 0
Viewed 111

I have approximately 1000 pieces of simulation data which have 2D array and size of the array is 1000(Row), 7(Col).

First, I tried to import and export the array just to see calculating speed and I programmed as below.

Input data structure

SIMULATION RESULTS
 0.599566E+00 0.666925E-06   0.3348E+02   0.2527E+03  -0.6948E+04  ...
 0.599633E+00 0.666924E-06   0.3394E+02   0.2529E+03  -0.6949E+04  ...
 0.599699E+00 0.666922E-06   0.3424E+02   0.2528E+03  -0.6949E+04  ...
 0.599766E+00 0.666920E-06   0.3440E+02   0.2527E+03  -0.6949E+04  ...
 0.599833E+00 0.666919E-06   0.3460E+02   0.2525E+03  -0.6948E+04  ...
 0.599899E+00 0.666919E-06   0.3488E+02   0.2522E+03  -0.6948E+04  ...
 0.599966E+00 0.666919E-06   0.3530E+02   0.2520E+03  -0.6948E+04  ...
.
.
.

first try

using Glob
filename = glob("*.dat*")
for i in filename
    data = readdlm(i, Float64, skipstart=1)
    writedlm("new_"*i, data)
end
close(data)

second try to keep specific format

using Glob
filename = glob("*.dat*")
for i in filename
    data = readdlm(i, Float64, skipstart=1)
    m = (a->(@sprintf "%15.3f" a)).(data)
    writedlm("new_"*i, m, "" , quotes=false)
end
close(data)

But the problem is that the calculating speed is very slow.

So, Please give me some advises so that I can deal with array data much faster (let me know if I don't

follow the Julia style and teach me some better way please.)

julia
1 Answers

You shouldn't be writing this manually. Try to use the extremely efficient CSV.jl package to read/write delimited files:

using DataFrames
using CSV

# read from disk
df = CSV.read("fname.dat", DataFrame, [options go here])

# do your calculations
df.result = df.column1 + df.column2

# write to disk
CSV.write("new.dat", df)

CSV.jl is the fastest reader/writer of delimited files across all popular languages. For more information read this old blog post: https://juliacomputing.com/blog/2020/06/fast-csv

And to correct a comment above that IO speed is similar in Julia and Python, the CSV.jl implementation beats Python's pandas by a factor of 10x to 20x.

user_25396230
Related