Perl multithreading slower when memory usage getting high

user_15322688 0
Viewed 118

Hi all~ I has written a very simple code in Perl using multithreading. The codes are as follows.

#!/bin/perl

use strict;
use threads;
use Benchmark qw(:hireswallclock);

my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = 1;
my $total_size = 10000000;
my $chunk_size = int($total_size / $num_of_threads);

if($total_size % $num_of_threads){
        $chunk_size++;
}

my @threads = ();

$starttime = Benchmark->new;

for(my $i = 0; $i < $num_of_threads; $i++) {
        my $thread = threads->new(\&search);
        push (@threads, $thread);
}

foreach my $thread (@threads) {
        $thread->join();
}

my $finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";

sub search{
        my $i = 0;
        while($i < $chunk_size){
            $i++;
        }

        return 1;
}

This piece of codes works fine as when increasing the number of threads, it will run faster.

However, when adding additional lines in the middle, which will create an array in big size, the code will run more slowly when adding more threads. The codes with the additional lines are shown below.

#!/bin/perl

use strict;
use threads;
use Benchmark qw(:hireswallclock);

my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = 1;
my $total_size = 10000000;
my $chunk_size = int($total_size / $num_of_threads);

if($total_size % $num_of_threads){
        $chunk_size++;
}

##########Additional codes##########
print "Preparing data...\n";
$starttime = Benchmark->new;

my @array = ();

for(my $i = 0; $i < $total_size; $i++){
        my $rn = rand();
        push(@array, $rn);
}

$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "Used ".timestr($timespent)."\n";
######################################

my @threads = ();

$starttime = Benchmark->new;

for(my $i = 0; $i < $num_of_threads; $i++) {
        my $thread = threads->new(\&search);
        push (@threads, $thread);
}

foreach my $thread (@threads) {
        $thread->join();
}

my $finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";

sub search{
        my $i = 0;
        while($i < $chunk_size){
            $i++;
        }

        return 1;
}

I am so confused regarding such a behaviour in Perl's multithreading. Does anyone has any idea what may go wrong here?

Thanks!

memory perl multithreading
2 Answers

You have to remember that when using ithreads - interpreter threads - the entire Perl interpreter, including code and memory, is cloned into the new thread. So the more data to be cloned, the longer it takes. There are ways to control what is cloned; take a look at the threads perldoc.

You should do as little as possible and not even load many modules until after you have spawned your threads.

If you do have a lot of data that will be used by all the threads, make it shared with threads::shared. Then to share a data structure use shared_clone(). You cannot simply share() anything, other than a simple variable. That shared variable can only contain plain scalars or other shared references.

If you are going to consume or pump that data, make it a queue instead with the Thread::Queue module. It automatically shares the values and takes care of the locking. After spawning your pool of worker threads, control them with Thread::Semaphore. That way they won't terminate before you've given them anything to do. You can also prevent race conditions.

https://metacpan.org/pod/threads::shared

HTH

user_114131060

Thank you all for pointing me to the relative directions! I have learned and tried different things, inlucding how to use shared and queue, which should be able to solve the problem. So I revised the script as follows:

#!/bin/perl

use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Benchmark qw(:hireswallclock);

my $starttime;
my $finishtime;
my $timespent;
my $num_of_threads = shift @ARGV;
my $total_size = 100000;
######Initiation of a 2D queue######
print "Preparing queue...\n";
$starttime = Benchmark->new;
my $queue = Thread::Queue->new();
for(my $i = 0; $i < $total_size; $i++){
        my $rn1 = rand();
        my $rn2 = rand();
        my @interval :shared = sort($rn1, $rn2);
        $queue->enqueue(\@interval);
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "Used ".timestr($timespent)."\n";
#####################################
$starttime = Benchmark->new;
my $queue_copy = $queue; #Copy the 2D queue so that the original queue can be kept\
for(my $i = 0; $i < $num_of_threads; $i++) {
        my $thread = threads->create(\&search, $queue_copy);
}
foreach my $thread (threads->list()) {
        $thread->join();
}
$finishtime = Benchmark->new;
$timespent = timediff($finishtime, $starttime);
print "$num_of_threads threads used in ".timestr($timespent)."\nDone!\n";
#####################################
sub search{
        my $temp_queue = $_[0];
        while(my $temp_interval = $temp_queue->dequeue_nb()){
                #Do something
        }
        return 1;
}

What I was trying to do was , first, to make a queue of arrays, with each containing two numbers. A copy of the queue was made as I want to preserve the original queue when going through it. Then the copied queue was gone through using multithreads. However, I still found out that it ran slower when more threads were added, which I have no clue why.

user_153226880
Related