Progress bar slows the loop

user_3281849 0
Viewed 227

I've done some tests with the progress bar and it slows down the test code considerably. Are there any alternatives or solutions? I'm looking for a way to track current index while looping and there are some primitive ways to put more conditions to print when step reached but isn't there something good that's built in? Oh and one more question, Is there a way to print time elapsed from when the function started and display with the index? let me clarify, I know about @time and etc but is there a way to count time and display it with corresponding index like "Reached index $i in iteration in time $time" Code for the tests done:

function test(x)
   summ = BigInt(0);
   Juno.progress(name = "foo") do id
       for i = 1:x
           summ+=i;
           @info "foo" progress=i/x _id=id
       end
   end
   println("sum up to $x is $summ");
   return summ;
end
@benchmark test(10^4)

function test2(x)
   summ = BigInt(0);
   for i = 1:x
       summ+=i;
       (i%10 == 0) && println("Reached this milestone $i")
   end
   println("sum up to $x is $summ");
   return summ;
end
@benchmark test2(10^4)

EDIT 1

for Juno.progress:

BenchmarkTools.Trial: 
memory estimate:  21.66 MiB
allocs estimate:  541269
--------------
minimum time:     336.595 ms (0.00% GC)
median time:      345.875 ms (0.00% GC)
mean time:        345.701 ms (0.64% GC)
maximum time:     356.436 ms (1.34% GC)
--------------
samples:          15
evals/sample:     1

For the crude simple version:

BenchmarkTools.Trial: 
memory estimate:  1.22 MiB
allocs estimate:  60046
--------------
minimum time:     111.251 ms (0.00% GC)
median time:      117.110 ms (0.00% GC)
mean time:        119.886 ms (0.51% GC)
maximum time:     168.116 ms (15.31% GC)
--------------
samples:          42
evals/sample:     1
julia
2 Answers

I'd recommend using Juno.@progress directly for much better performance:

using BenchmarkTools

function test(x)
    summ = BigInt(0)
    Juno.progress(name = "foo") do id
        for i = 1:x
            summ += i
            @info "foo" progress = i / x _id = id
        end
    end
    println("sum up to $x is $summ")
    return summ
end
@benchmark test(10^4) # min: 326ms

function test1(x)
    summ = BigInt(0)
    Juno.@progress "foo" for i = 1:x
        summ += i
    end
    println("sum up to $x is $summ")
    return summ
end
@benchmark test1(10^4) # min 5.4ms

function test2(x)
    summ = BigInt(0)
    for i = 1:x
        summ += i
    end
    println("sum up to $x is $summ")
    return summ
end
@benchmark test2(10^4) # min 0.756ms


function test3(x)
   summ = BigInt(0);
   for i = 1:x
       summ+=i;
       (i%10 == 0) && println("Reached this milestone $i")
   end
   println("sum up to $x is $summ");
   return summ;
end
@benchmark test3(10^4) # min 33ms

Juno.progress can make no performance optimizations at all for you, but you can implement them manually:

function test4(x)
    summ = BigInt(0)
    update_interval = x÷200 # update every 0.5%
    Juno.progress(name = "foo") do id
        for i = 1:x
            summ += i
            if i % update_interval == 0
                @info "foo" progress = i / x _id = id
            end
        end
    end
    println("sum up to $x is $summ")
    return summ
end
@benchmark test4(10^4) # min: 5.2ms
user_121131780

As was stated by High Performance Mark writing to the screen is fundamentally slow (crazy fast in human scale, very slow in computer scale.) You could abandon writing the output to the progress bar, but you can also simply update the progress bar less often. In your test case you're doing 10000 additions and updating the progress bar 10000 times. To be honest I've never used Julia and I have no idea what the progress bar looks like. Even if it is a GUI progress bar on a 4K screen and each of these updates actually changes it at all I guarantee a human can't see the difference. I would update it at the beginning (to be 0) and at the end (to be 100%) and then use an if statement with a modulo test to only update every so many additions. Example below in python which I'll claim is pseudo code since I've never used julia:

updateEvery = 2
for i in range(1,x):
    sum += i
    if x % updateEvery == 0:
        updateProgressBar(i/x)

By varying updateEvery you can decrease or increase the number of progress bar updates. You can even calculate it dynamically based on x, say updateEvery = x/100, this would mean the progress bar would line up pretty well to percentages. The inefficiency caused by the progress bar updates is also probably meaningless for small values of x and as x increases the number of updates per number to be added will decrease (because the total number of updates will be constant.

Oh and if you really need great performance to the counting clock tick level (which you probably don't,) modulo is faster for powers of 2 as it can be done with a binary and operation. I assume Julia will figure this optimisation out for you and you can just use % and round the value of updateEvery to the next power of 2. Though if you really care about that level of performance you'd be best to just get rid of the progress bar to eliminate the loop altogether.

user_119903170
Related