Is there a fast parallel "for" loop in Perl 6?

user_1682146 0
Viewed 502

Given some code which does a bit of math/casting for each number from 1 to 500000, we have options:

  1. Simple for loop: for ^500000 -> $i { my $result = ($i ** 2).Str; }. In my unscientific benchmark, this takes 2.8 seconds.

  2. The most canonical parallel version does each bit of work in a Promise, then waits for the result. await do for ^500000 -> $i { start { my $result = ($i ** 2).Str; } } takes 19 seconds. This is slow! Creating a new promise must have too much overhead to be worthwhile for such a simple computation.

  3. Using a parallel map operation is fairly fast. At 2.0 seconds, the operation seems barely slow enough to take advantage of parallelization: (^500000).race.map: -> $i { my $result = ($i ** 2).Str; }

The third option seems best. Unfortunately, it reads like a hack. We should not be writing map code for iteration in sink context, because others that read "map" in the source may assume the purpose is to build a list, which isn't our intent at all. It's poor communication to use map this way.

Is there any canonical fast way to use Perl 6's built in concurrency? A hyper operator would be perfect if it could accept a block instead of only functions:

(^500000)».(-> $i { my $result = ($i ** 2).Str; }) # No such method 'CALL-ME' for invocant of type 'Int'
concurrency raku
2 Answers

If you want to use for with a hyper or race operation, you have to spell it hyper for @blah.hyper(:batch(10_000)) or race for @blah.race(:batch(10_000)). Or without parameters: hyper for @blah, race for @blah.

This was decided because you might have code like for some-operation() { some-non-threadsafe-code } where some-operation is part of a library or something. Now you cannot tell any more if the for loop can have thread-unsafe code in it or not, and even if you know the library doesn't return a HyperSeq at that point in time, what if the library author comes up with this great idea to make some-operation faster by hypering it?

That's why a signifier for "it's safe to run this for loop in parallel" is required right where the code is, not only where the sequence gets created.

user_8040050

On my PC, this is a bit (~15%) faster than the naive loop:

(^500_000).hyper(batch => 100_000).map(-> $i { my $result = ($i ** 2).Str; })

Since the computation inside the loop is really fast, typically the cost of parallelization and synchronization dwarfs any gains you get from it. The only remedy is a large batch size.

Update: with a batch size of 200_000 I get slightly better results (another few percent faster).

user_141320
Related