Combining vectors of unequal length

Viewed 122
x = [1, 2, 3, 4]
y = [1, 2]

If I want to be able to operate on the two vectors with a default value filling in, what are the strategies?

E.g. would like to do the following and implicitly fill in with 0 or missing

x + y        # would like [2, 4, 3, 4] 

Ideally would like to do this in a generic way so that I could do arbitrary operations with the two.

2 Answers

Disregarding whether Julia has something built-in to do this, remember that Julia is fast. This means that you can write code to support this kind of need.

extend!(x, y::Vector, default=0) = extend!(x, length(y), default)
extend!(x, n::Int, default=0) = begin
    while length(x) < n
        push!(x, default)
    end
    x
end

Then when you have code such as you describe, you can symmetrically extend x and y:

x = [1, 2, 3, 4]
y = [1, 2]
extend!(x, y)
extend!(y, x)
x + y
==> [2, 4, 3, 4]

Note that this mutates y. In many cases, the desired length would come from outside the code and would be applied to both x and y. I can also imagine that 0 is a bad default in general (even though it is completely appropriate in your context of addition.

A comment below makes the worthy point that you should consider using append! instead of looping over push!. In fact, it is best to measure differences like that if you care about very small differences. I went ahead and tested:

julia> using BenchmarkTools
julia> extend1(x, n) = begin
       while length(x) < n
           push!(x, 0)
       end
       x
end
julia> @btime begin
       x = rand(10)
       sum(x)
       end
  59.815 ns (1 allocation: 160 bytes)
5.037723569560573

julia> @btime begin
       x = rand(10)
       extend1(x, 1000)
       sum(x)
       end
  7.281 μs (8 allocations: 20.33 KiB)
6.079832879992913
julia> x = rand(10)
julia> @btime begin
       x = rand(10)
       append!(x, zeros(990))
       sum(x)
       end
  1.290 μs (3 allocations: 15.91 KiB)
3.688526541987817
julia> 

Pushing primitives in a loop is damned fast, allocating a vector of zeros so we can use append! is very slightly faster.

But the real lesson here is seen in the fact that the loop version takes microseconds to append nearly 1000 values (reallocating the array several times). Appending 10 values one by one takes just over 150ns (and append! is slightly faster). This is blindingly fast. Literally doing nothing in R or Python can take longer than this.

This difference would matter in some situations and would be undetectable in many others. If it matters, measure. If it doesn't, do the simplest thing that comes to mind because Julia has your back (performance-wise).

FURTHER UPDATE

Taking a hint from another of Colin's comments, here are results where we use append! but we don't allocate a list. Instead, we use a generator ... that is, a data structure that invents data when asked for it with an interface much like a list. The results are much better than what I showed above.

julia> @btime begin
       x = rand(10)
       append!(x, (0 for i in 1:990))
       sum(x)
       end
  565.814 ns (2 allocations: 8.03 KiB)

Note the round brackets around the 0 for i in 1:990.

In the end, Colin was right. Using append! is much faster if we can avoid related overheads. Surprisingly, the base function Iterators.repeated(0, 990) is much slower.

But, no matter what, all of these options are pretty blazingly fast and all of them would probably be so fast that none of these subtle differences would matter.

Julia is fun!

Note that if you want to fill with missing or some other type different from the element type in your original vector, then you will need to change the type of your vectors to allow those new elements. The function below will handle any case.

function fillvectors(x, y, fillvalue=missing)
    xl = length(x)
    yl = length(y)
    if xl < yl
        x::Vector{Union{eltype(x), typeof(fillvalue)}} = x
        for i in xl+1:yl
            push!(x, fillvalue)
        end
    end
    if yl < xl
        y::Vector{Union{eltype(y), typeof(fillvalue)}} = y
        for i in yl+1:xl
            push!(y, fillvalue)
        end
    end
    return x, y
end

x = [1, 2, 3, 4]
y = [1, 2]
julia> (x, y) = fillvectors(x, y)
([1, 2, 3, 4], Union{Missing, Int64}[1, 2, missing, missing])

julia> y
4-element Vector{Union{Missing, Int64}}:
 1
 2
  missing
  missing
julia> (x, y) = fillvectors(x, y, 0)
([1, 2, 3, 4], [1, 2, 0, 0])

julia> y
4-element Vector{Int64}:
 1
 2
 0
 0
julia> (x, y) = fillvectors(x, y, 1.001)
([1, 2, 3, 4], Union{Float64, Int64}[1, 2, 1.001, 1.001])

julia> y
4-element Vector{Union{Float64, Int64}}:
 1
 2
 1.001
 1.001
Related