Saving arrays of different length in a single data frame in Julia

user_3935797 0
Viewed 1292

I want to save results of a simulation that outputs an array of variable length. Normally I would run the simulation, save it in an array, concatenate it with an array with previous results and then save the array which has all results by making a data frame and then using CSV.write. However since the arrays are of variable length hcat() won't work. Below a toy example of what I would like to do.

output = zeros(5)
number_simulations = 10
for i = 1:number_simulations
    l = sample([4, 5, 6, 7])
    print(l)
    for j = 1:l
        new_out = zeros(l)
        hcat(output, new_out)
    end    
end
df = convert(DataFrame, output)
CSV.write("out.csv", df)

This returns error

DimensionMismatch("vectors must have same lengths").

Is there a simple workaround that would allow me to have a file with the results of each simulation in separate columns?

julia arrays
1 Answers

The best practice here would probably be to just use a "tall"/"long"/"tidy" dataset in which you store the simulation number in one vector and the vertically stacked results in another (and possibly the indices in a third).

But to achieve what you're after, I'd just store the output directly into a vector of DataArrays. Then you can resize them to the size of the largest one at the end:

julia> output = DataVector{Float64}[]
       number_simulations = 10
       for i = 1:number_simulations
           l = sample([4, 5, 6, 7])
           print(l)
           for j = 1:l
               push!(output, zeros(l)) # This converts the array to a DataVector
           end
       end
5746656565
julia> resize!.(output, maximum(length, output))
       DataFrame(output)
7×55 DataFrames.DataFrame
│ Row │ x1  │ x2  │ x3  │ x4  │ x5  │ x6  │ x7  │
├─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┤
│ 1   │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │
│ 2   │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │
│ 3   │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │
│ 4   │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │
│ 5   │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │
│ 6   │ NA  │ NA  │ NA  │ NA  │ NA  │ 0.0 │ 0.0 │
│ 7   │ NA  │ NA  │ NA  │ NA  │ NA  │ 0.0 │ 0.0 │

⋮
user_1760710
Related