preventing extreme memory usage

user_3472157 0
Viewed 134

my script works on a line by line basis. The task per line implies use of a Grammar, but is not too complicated. I notice that when the input has a lot of lines, say 150_000, the memory usage is gigantic, over 6G, and my computer hangs. I used --profile on shorter input, but that gives me no clue where to look for a solution. Most names in there do not refer to my code. Any suggestions on where I can prevent spurious memory use or find out what is causing this?

here is the core loop:

method process() {
    my $promise = Promise.start({
      &!start-handler();
      my $intro = 1; # to skip the directory part of the file.
      for $!source-name.IO.lines {
        ...
      }
    }).then({ $!DB.completed });
    await $promise;
  }

Thanks, Theo

raku memory
1 Answers

I found one thing that caused havoc. By changing the line

m:g/ << $<funname>=<[A..Z]>+ '(' <{ %fun{$<funname>}++ }> /;

by something like

 if $formula ~~ m:g/ << $<funname>=<[A..Z]>+  '(' / {
        for $/.list -> $funname {
          %fun{$<funname>}++
        };
      }

the script gets a zillion times faster and runs without problems.

However, see @JonathanWorthington's reaction below. My conclusion: I should have used:

m:g/ << $<funname>=<[A..Z]>+ '(' { %fun{$<funname>}++ } /;

Again, this used to work in previous editions of rakudo. This is Rakudo version 2020.01 built on MoarVM version 2020.01.1 implementing Perl 6.d.

user_34721570
Related