Performance while reading 2 files in parallel in perl

user_3142647 0
Viewed 138

I need to join 2 big files. The task is similar to what the join utility is doing. While playing around with my algorithm, which is very similar to this problem: I've realized that coreutil's join is way faster ( but not doing exactly what I need). However after checking this perl code for join implemented in perl which seem to be quite similar to the C code of the coreutil's join I see that the difference is the same, perl code is too slow.

Some tests:

time join -j 2 file1.txt file2.txt| mbuffer > result.c
in @ 20.0 MiB/s, out @ 20.0 MiB/s, 86.0 MiB total, buffer   0% full
summary: 95.6 MiByte in  5.9sec - average of 16.2 MiB/s
join -j 2 file1.txt file2.txt  4.99s user 0.09s system 99% cpu 5.091 total

time perl ./join -j 1 file1.txt file2.txt| mbuffer > result.perl                                                                                   
in @ 4092 kiB/s, out @ 4092 kiB/s, 94.0 MiB total, buffer   0% full
summary: 95.6 MiByte in 44.5sec - average of 2202 kiB/s
perl ./join -j 1  file1.txt file2.txt  44.15s user 0.08s system 99% cpu 44.226 total

Any hint on how to improve the performance? I suspect it might be related to the buffering.

According to nytrpof the bottleneck is get_a_line() sub

# spent 59.4s (57.3+2.12) within main::get_a_line which was called 25884852 times, avg 2µs/call:
# 15574181 times (34.4s+1.27s) by main::RUNTIME at line 69, avg 2µs/call
# 10309895 times (22.9s+857ms) by main::RUNTIME at line 75, avg 2µs/call
#      478 times (868µs+42µs) by main::RUNTIME at line 85, avg 2µs/call
#      296 times (533µs+30µs) by main::RUNTIME at line 80, avg 2µs/call
#           once (12µs+26µs) by main::RUNTIME at line 59
#           once (4µs+7µs) by main::RUNTIME at line 60
sub get_a_line {
121 25884852    2.60s             my ($aref, $fh) = @_;
122 25884852    29.1s   25884852    2.12s     my $not_eof = defined(my $line = <$fh>);
  # spent  2.12s making 25884852 calls to main::CORE:readline, avg 82ns/call
123 25884852    5.50s             if ($not_eof) {
124 25884851    1.92s               chomp $line;
125 25884851    12.5s               push (@$aref,
126                           defined $delimiter ?
127                           [split $delimiter, $line, -1] : [split ' ', $line, -1]);
128                   }
129 1   300ns             else { push @$aref, undef }
130 25884852    27.4s             return $not_eof;
131                 }
io perl
1 Answers

As suggested I did a minimal test and got much better results:

#!/usr/bin/perl
use strict;
use warnings;

for (@ARGV) { die "need 2 files" if ! -e };
my ($file1,$file2) = @ARGV;

sub read_f {
    my $file = shift;
    open my $fh, ($file =~ m/\.gz/ ? "gzip -fdc $file|mbuffer|" : $file)
        or die "Can't open file $file: $!\n";
    return $fh;
}

sub read_file_line {
    my $fh = shift;
    if ($fh and my $line = readline $fh) {
        return $line;
    }
    return;
}

my $f1 = read_f($file1);
my $f2 = read_f($file2);

my $line1 = read_file_line($f1);
my $line2 = read_file_line($f2);

while ($line1 or $line2) {
    $line1 = read_file_line($f1);
    $line2 = read_file_line($f2);
}

the result (mbuffer measures different things for perl and c code- reading and writing, but the global picture is clean enough I guess):

time perl ./testcase.pl sample1.gz sample2.gz
in @  0.0 kiB/s, out @ 20.0 MiB/s,  164 MiB total, buffer   2% full                                                                                                                
summary:  172 MiByte in  8.5sec - average of 20.2 MiB/s
in @  0.0 kiB/s, out @ 12.0 MiB/s,  202 MiB total, buffer   1% full
summary:  207 MiByte in 11.4sec - average of 18.1 MiB/s
perl ./testcase.pl sample1.gz sample2.gz 12.75s user 2.23s system 130% cpu 11.462 tota

time join -j 2 -a 1 -a 2  <(zcat sample.gz) <(sample2.gz ) |mbuffer 1>/dev/null 
in @ 40.0 MiB/s, out @ 40.0 MiB/s,  326 MiB total, buffer   0% full

summary:  330 MiByte in  6.8sec - average of 48.6 MiB/s
join -j 2 -a 1 -a 2 <(zcat sample1.gz)   6.55s user 0.24s system 99% cpu 6.789 total
mbuffer > /dev/null  0.04s user 0.44s system 7% cpu 6.799 total

So it appears that additional operations like split,join, callings subs and passing data back and forth add a lot of time. And since the data is really huge it all sum-ups...

user_31426470
Related