How to benchmark Raku?

user_2774153 0
Viewed 273

I regularly follow the GitHub Rakudo repository to see what changes are going on in the Rakudo compiler.

I sometimes see commits where the individual functions are sped up by certain percentage, and times like the image below.

enter image description here

What is the workflow to evaluate this? I am keen to learn this so one can know how your functions are performing and accordingly optimize further and can contribute to Rakudo development.

I read for help here, here. I googled but could not find this information. I also learnt about MoarVM profiler via command line --profile option here which generates html output. What is it to look for?

enter image description here

I am not a formal computer science guy. I understand famous quote of Tony Hoare — “Premature optimization is the root of all evil” but with passage of time, once the code is written correct, one wishes to optimize it. So the question.

rakudo benchmarking raku
1 Answers

I use --profile to get a better idea of where the bottlenecks are. The produced profile is a good start, but not very good for CPU usage when the differences become very small. It is however pretty good at tracking allocations of objects, and fewer object allocations at least can mean less memory churn (not always though, if the object are very short-lived). And the keeping track of stuff with --profile has its effects on optimizations as well, so Heisenberg's uncertainty principle definitely applies here.

Once I have a piece of code of before / after, I run it either as a script or as a one liner with time. I have a bunch of handy aliases that help me with that:

alias r='time raku -e'
alias rp='raku --profile -e'

The reason I do it as separate processes with at least a few seconds inbetween, is that:

  1. running multiple benchmarks in the process tend to heat up the CPU, which will then get downthrottled, making the later benchmark worse.
  2. if both benchmarks share some code in the core, the later benchmark may benefit from that code having been inlined / JITted by the earlier benchmark.

I then run each of the before and after code 3 to 5 times, and a Nil loop to find out the overhead. So e.g.:

$ r 'my $a = "42"; Int($a) for ^100000'
real    0m0.244s

$ r 'my $a = "42"; $a.Int for ^100000'
real    0m0.178s

$ r 'my $a = "42"; Nil for ^100000'
real    0m0.154s

And then calculate the difference:

$ r 'say (244 - 154) / (178 - 154)'
3.75

So it's about 3.75x as fast to use $a.Int than Int($a). Which of course could start another --profile cycle finding out why Int($a) is so much slower. Also, when I see differences in speed I cannot explain, I use a --profile to find out if it's really doing the things I think it's doing. Specifically unexpected constant-folding can sometimes make you think you found the optimal optimization, when in fact you reduced your code to doing basically nothing.

HTH

user_74244700
Related