BenchmarkTools outputs to DataFrame

Question

I am trying to benchmark the performance of functions using BenchmarkTools as in the example below. My goal is to obtain the outputs of @benchmark as a DataFrame.

In this example, I am benchmarking the performance of the following two functions:

"""Example function A: recodes negative values to 0"""
function negative_to_zero_a!(x::Array{<:Real,1}) 
    for (i, v) in enumerate(x)
        if v < 0
            x[i] = zero(x[i]) # uses 'zero()'
        end 
    end
end 

"""Example function B: recodes negative values to 0"""
function negative_to_zero_b!(x::Array{<:Real,1}) 
    for (i, v) in enumerate(x)
        if v < 0
            x[i] = 0 # does not use 'zero()'
        end 
    end
end

Which are meant to mutate the following vectors:

int_a = [1, -2, 3, -4]
float_a = [1.0, -2.0, 3.0, -4.0]

int_b = copy(int_a)
float_b = copy(float_a)

I then produce the performance benchmarks using BenchmarkTools.

using BenchmarkTools

int_a_benchmark = @benchmark negative_to_zero_a!(int_a)
int_b_benchmark = @benchmark negative_to_zero_b!(int_b)

float_a_benchmark = @benchmark negative_to_zero_a!(float_a)
float_b_benchmark = @benchmark negative_to_zero_b!(float_b)

I would now like to retrieve the elements of each of the four BenchmarkTools.Trial objects into a DataFrame similar to the one below. In that DataFrame, each row contains the results of a given BenchmarkTools.Trial object. E.g.

DataFrame("id" => ["int_a_benchmark", "int_b_benchmark", "float_a_benchmark", "float_b_benchmark"],  
          "minimum" => [15.1516, 15.631, 14.615, 14.271], 
          "median" => [15.916, 15.731, 15.916, 15.879], 
          "maximum" => [149.15, 104.108, 63.363, 116.181],
          "allocations" => [0, 0, 0, 0],
          "memory_bytes" => [0, 0, 0, 0])

4×6 DataFrame
 Row │ id                 minimum  median   maximum  allocations  memory_estimate 
     │ String             Float64  Float64  Float64  Int64        Int64
─────┼────────────────────────────────────────────────────────────────────────────
   1 │ int_a_benchmark    15.1516   15.916  149.15             0                0
   2 │ int_b_benchmark    15.631    15.731  104.108            0                0
   3 │ float_a_benchmark  14.615    15.916   63.363            0                0
   4 │ float_b_benchmark  14.271    15.879  116.181            0                0

How can I retrieve the results of the benchmarks into a DataFrame like this one?

user_1269567 · Answer

You can do it e.g. like this:

julia> using Statistics, DataFrames, BenchmarkTools

julia> preprocess_trial(t::BenchmarkTools.Trial, id::AbstractString) =
           (id=id,
            minimum=minimum(t.times),
            median=median(t.times),
            maximum=maximum(t.times),
            allocations=t.allocs,
            memory_estimate=t.memory)
preprocess_trial (generic function with 1 method)

julia> output = DataFrame()
0×0 DataFrame

julia> for (fun, id) in [(sin, "sin"), (cos, "cos"), (log, "log")]
           push!(output, preprocess_trial(@benchmark(sin(1)), id))
       end

julia> output
3×6 DataFrame
 Row │ id      minimum  median   maximum  allocations  memory_estimate
     │ String  Float64  Float64  Float64  Int64        Int64
─────┼─────────────────────────────────────────────────────────────────
   1 │ sin       0.001    0.001      0.1            0                0
   2 │ cos       0.001    0.001      0.1            0                0
   3 │ log       0.001    0.001      0.1            0                0

user_1494135 · Answer

As usual with Julia, there are multiple ways to do what you want. I present here maybe not the simplest way, but the one, which hopefully shows interesting approach, which allows for some generalizations.

But before we start, small notice: your benchmarks are not quite correct, since your functions mutate argument. For proper benchmarking you should copy your data before each run and also do it every time you execute function. You can find more information here: https://juliaci.github.io/BenchmarkTools.jl/dev/manual/#Setup-and-teardown-phases

So, for now, we assume that you prepared benchmarks like this

int_a_benchmark = @benchmark negative_to_zero_a!(a) setup=(a = copy($int_a)) evals=1
int_b_benchmark = @benchmark negative_to_zero_b!(b) setup=(b = copy($int_b)) evals=1

float_a_benchmark = @benchmark negative_to_zero_a!(a) setup=(a = copy($float_a)) evals=1
float_b_benchmark = @benchmark negative_to_zero_b!(b) setup=(b = copy($float_b)) evals=1

Main idea is the following. If we can represent our benchmark data as a DataFrame, then we can combine them together as a single large DataFrame and do all necessary calculations.

Of course one can do it in super easy way, just by making command

df = DataFrame(times = int_a_benchmark.times, gctimes = int_a_benchmark.gctimes)
df.memory .= int_a_benchmark.memory
df.allocs .= int_a_benchmark.allocs

but this is too boring and too verbatim (but simple and should be done 99% of time). It would be nice, if we can just do DataFrame(int_a_benchmark) and get the result immediately.

As it turns out, it is possible, because DataFrames supports Tables.jl interface for working with table-like data. You can read details in the manual of Tables.jl, but generally you need to define some meaningful things, like names of columns, and column accessors and package will do everything else. I show the results here, without further explanations.

using Tables

Tables.istable(::Type{<:BenchmarkTools.Trial}) = true
Tables.columnaccess(::Type{<:BenchmarkTools.Trial}) = true
Tables.columns(m::BenchmarkTools.Trial) = m
Tables.columnnames(m::BenchmarkTools.Trial) = [:times, :gctimes, :memory, :allocs]
Tables.schema(m::BenchmarkTools.Trial) = Tables.Schema(Tables.columnnames(m), (Float64, Float64, Int, Int))
function Tables.getcolumn(m::BenchmarkTools.Trial, i::Int)
    i == 1 && return m.times
    i == 2 && return m.gctimes
    i == 3 && return fill(m.memory, length(m.times))
    return fill(m.allocs, length(m.times))
end
Tables.getcolumn(m::BenchmarkTools.Trial, nm::Symbol) = Tables.getcolumn(m, nm == :times ? 1 : nm == :gctimes ? 2 : nm == :memory ? 3 : 4)

and we can see that it really works (almost magically)

julia> DataFrame(int_a_benchmark)
10000×4 DataFrame
   Row │ times    gctimes  memory  allocs
       │ Float64  Float64  Int64   Int64
───────┼──────────────────────────────────
     1 │   309.0      0.0       0       0
     2 │    38.0      0.0       0       0
     3 │    25.0      0.0       0       0
     4 │    37.0      0.0       0       0
   ⋮   │    ⋮        ⋮       ⋮       ⋮

Next step is combining all dataframes in a single dataframe. We should make following steps:

Convert benchmark trial to dataframe
Add column name with the name of relevant benchmark
Group them all together (with vcat function)

Of course, you can do all of this steps one by one for each dataframe, but it's too long (and boring, yes). Instead, we can use amazing mapreduce function and so called Do-Block syntax. map part will prepare necessary dataframes and reduce will combine them together

df_benchmark = mapreduce(vcat, zip([int_a_benchmark, int_b_benchmark, float_a_benchmark, float_b_benchmark],
        ["int_a_benchmark", "int_b_benchmark", "float_a_benchmark", "float_b_benchmark"])) do (x, y)
    df = DataFrame(x)
    df.name .= y
    df
end

And now for the final part. We have nice, large DataFrame, which we want to aggregate. For this we can use Split-Apply-Combine strategy of DataFrames

julia> combine(groupby(df_benchmark, :name), 
                :times => minimum => :minimum,
                :times => median => :median,
                :times => maximum => :maximum,
                :allocs => first => :allocations,
                :memory => first => :memory_estimate)

4×6 DataFrame
 Row │ name               minimum  median   maximum  allocations  memory_estimate
     │ String             Float64  Float64  Float64  Int64        Int64
─────┼────────────────────────────────────────────────────────────────────────────
   1 │ int_a_benchmark       22.0     24.0   3252.0            0                0
   2 │ int_b_benchmark       20.0     23.0    489.0            0                0
   3 │ float_a_benchmark     21.0     23.0    134.0            0                0
   4 │ float_b_benchmark     21.0     23.0    129.0            0                0

As a bonus, last calulcation can look even better with the help of Chain.jl package:

using Chain

@chain df_benchmark begin
    groupby(:name)
    combine(:times => minimum => :minimum,
            :times => median => :median,
            :times => maximum => :maximum,
            :allocs => first => :allocations,
            :memory => first => :memory_estimate)
end

BenchmarkTools outputs to DataFrame

2 Answers