Perl Automating Hash Population and Key Name

Viewed 142

I have a number of error strings. I am matching them to patterns I already have. If they're the exact same I want them to fall under the exact same error failure. If they match the pattern but have some difference from previous strings in the hash, I want to give it the same error name but with a different number appended to it.

Here's an example input file:

there are 5 syntax issues with semicolon
there are 11 syntax issues with semicolon
the file contains 5 formatting issues
there are 1 syntax issues with semicolon
check script for formatting issues
2 syntax issues have been found
the file contains 1 formatting issues
6 syntax issues have been found
use warnings;
use strict;

my %errors;
my $file = "listoferrormessages.txt"

open my $fh,'<',$file or die "Could not open $file: $!";

while(my $line = <$fh>){

if( $line =~ /syntax/){

    if ($line =~ /there are \d syntax issues with semicolon/){
       #if line matching format exists in hash values, continue
       #if not, create a hash key called syntax_# where # increments one from the last key with the error name. 
        $errors{errorname} = $line;
}

    elsif ($line =~ /\d syntax issues have been found/){
       #same as above
       $errors(errorname} = $line;
}

elsif ($line =~ /format/){
#same as above
}

}
close $fh;

I would like my hash to look like:

$VAR1 = {
          'syntax_1' => 
                     'there are 5 syntax issues with semicolon',
          'syntax_2' => 
                     '2 syntax issues have been found',
          'format_1' => 
                     'the file contains 5 formatting issues',
          'format_2' => 
                     'check script for formatting issues'
        };

Any guidance on this would be super helpful. There's a lot more I'd like to add on but I'm very confused with how I can start doing this. Is this even possible to do?

3 Answers

This does what is asked, with a remaining question of possible error types.

An auxiliary data structure (%seen_error_type) is there to avoid searching through values on every line, to check whether that error-type has been seen; with this hash it's just a lookup.

use warnings;
use strict;
use feature qw(say);

use Data::Dump qw(dd);  # to show complex data structures

my $file = shift // die "Usage: $0 file\n";  #/
open my $fh, '<', $file  or die "Can't open $file: $!";

my (%error, %seen_error_type, $cnt_syntax, $cnt_format);

LINE:
while (my $line = <$fh>) { 
    chomp $line;

    my $error_type = $line =~ s/[0-9]+/N/r;  # extract error type

    next LINE if exists $seen_error_type{$error_type};
    $seen_error_type{$error_type} = 1;

    if ($line =~ /syntax/) {
        ++$cnt_syntax;
        $error{ "syntax_$cnt_syntax" } = $line;
    }
    elsif ($line =~ /format/) {
        ++$cnt_format;
        $error{ "format_$cnt_format" } = $line;
    }   
    else { }  # specify how to handle unexpected error types
}       
    
dd \%error;

An error "type" is first built from a line, by replacing a number with N; this merely follows OP samples since no rule is given for how to categorize those error messages. If that's indeed all that there is, fine. But I'd expect more complex criteria for kinds of errors to expect.

The key need for improving this is to articulate rules for what "error types" (structure of error messages) are expected.

Simply adding unexpected patterns to our bookkeeping hash of error types doesn't makes sense unless we have some rule for how to extract a pattern out of a line. Otherwise each possible line of text may end up being a key for itself, which would defeat the purpose of the whole exercise of classifying them.

With the given input file the above prints

{
  format_1 => "the file contains 5 formatting issues",
  format_2 => "check script for formatting issues",
  syntax_1 => "there are 5 syntax issues with semicolon",
  syntax_2 => "2 syntax issues have been found",
}

(The Data::Dump module I used probably need be installed. A core option is Data::Dumper)


Another note, raised in comments: I don't see why to add a key for each new line, instead of adding each expected error-type-line to an arrayref for a suitable key (syntax, format, etc).

If there is no specific reason for that then I'd rather suggest something like

my (%error, %seen_error_type);

LINE:
while (my $line = <$fh>) { 
    chomp $line;

    my $error_type = $line =~ s/[0-9]+/N/r;  # extract error type

    next LINE if exists $seen_error_type{$error_type};
    $seen_error_type{$error_type} = 1;

    if ($line =~ /syntax/) {
        push @{$error{syntax}}, $line;
    }   
    elsif ($line =~ /format/) { 
        push @{$error{format}}, $line;
    }
    else { }  # specify how to handle unexpected error types
}

dd \%error;

Now we simply have an array reference for key syntax, and another for key format.

This prints

{
  format => [
              "the file contains 5 formatting issues",
              "check script for formatting issues",
            ],
  syntax => [
              "there are 5 syntax issues with semicolon",
              "2 syntax issues have been found",
            ],
}

zdim's program is workable, but if your situation was more complicated that while loop is going to get messy. There's a better pattern you can use so you can continue to add patterns. Polar Bear got close, but still has extra, special knowledge baked into the loop.

Create a table of the sorts of things that you want to match. Moving this information out of the loop has several advantages. First, it makes the loop simpler. Second, it's easier to see parallel structure. Third, this is one step closer to storing the matcher information in configuration. The order in the array is the order that I want to test them:

my @matchers = (
    #  label    pattern
    [ 'syntax', qr/syntax/ ],
    [ 'format', qr/format/ ],
    );

The loop then becomes something like this. This version of the while works for as many matchers as you care to define. This while has no special knowledge about the input or matching. It's job is to digest all the data into something that you can easily manage later, and to do it without losing information:

use v5.26;
my %hash;

LINE:
while (my $line = <$fh>) {
    chomp $line;

    foreach my $matcher ( @matchers ) {
        next unless $line =~ m/$matcher->[1]/;
        my( $n ) = $line =~ /(\d+)/;
        push $hash{ $matcher->[0] }{$n // 0}->@*, $line;
        next LINE; # or not, if you want to sort into multiple categories
        }
    }

Now I have a data structure that has the number in the error message and a list of all the lines that matched. This doesn't have to be your final data structure, but it can get you there. Eventually the requirements from the output will change and if you've baked too many of the decisions into the while, you'll have to start over. Instead, you end up with this no matter the output and it's only a matter of converting this to whatever you want to end up with. I would certainly want to see how different lines sort into categories. I can quickly track down a miscategorization this way:

{
  format => {
              "0" => ["check script for formatting issues"],
              "1" => ["the file contains 1 formatting issues"],
              "5" => ["the file contains 5 formatting issues"],
            },
  syntax => {
              1  => ["there are 1 syntax issues with semicolon"],
              2  => ["2 syntax issues have been found"],
              5  => [
                      "there are 5 syntax issues with semicolon",
                      "there are 5 syntax issues with commas",
                    ],
              6  => ["6 syntax issues have been found"],
              11 => ["there are 11 syntax issues with semicolon"],
            },
}

I'd be tempted to go one step further to include the line number with each line. The line number of the last read file handle is the special var $.:

push $hash{ $matcher->[0] }{$n // 0}->@*, [$line, $.];

The data structure now has an extra level of structure. I could already tell the order of lines within a label and count, but now I know the order of lines across the entire data structure. If I wanted, I could recreate the input:

{
  format => {
              "0" => [["check script for formatting issues", 6]],
              "1" => [["the file contains 1 formatting issues", 8]],
              "5" => [["the file contains 5 formatting issues", 4]],
            },
  syntax => {
              1  => [["there are 1 syntax issues with semicolon", 5]],
              2  => [["2 syntax issues have been found", 7]],
              5  => [
                      ["there are 5 syntax issues with semicolon", 1],
                      ["there are 5 syntax issues with commas", 2],
                    ],
              6  => [["6 syntax issues have been found", 9]],
              11 => [["there are 11 syntax issues with semicolon", 3]],
            },
}

Looking at the input data I see repeating pattern /\d+ (syntax|formatting) issues/ which gives us a clue about type problem we look at.

Why do not utilize it to split issues into groups base on the type?

use strict;
use warnings;
use feature 'say';

use Data::Dumper;

my $regex = qr/\d+ (syntax|formatting) issues/;
my $issues;

while( <DATA> ) {
    chomp;
    next unless /$re/;

    my $type = $1;
    $type = 'format' if $type =~ /formatting/;

    push @{$issues->{$type}}, $_;
}

say Dumper($issues);

__DATA__
there are 5 syntax issues with semicolon
there are 11 syntax issues with semicolon
the file contains 5 formatting issues
there are 1 syntax issues with semicolon
check script for formatting issues
2 syntax issues have been found
the file contains 1 formatting issues
6 syntax issues have been found

Output

$VAR1 = {
          'format' => [
                        'the file contains 5 formatting issues',
                        'the file contains 1 formatting issues'
                      ],
          'syntax' => [
                        'there are 5 syntax issues with semicolon',
                        'there are 11 syntax issues with semicolon',
                        'there are 1 syntax issues with semicolon',
                        '2 syntax issues have been found',
                        '6 syntax issues have been found'
                      ]
        };
Related