How to declare and assign a hash with count of occurrences in one statement

Question

I'm working on an NLP project for automatic text classification. Here is a snippet of a code to constitute a primitive bag of words from an XML file. But my question is as follows: in this sequence, is it possible to one-line these 3 lines because i don't really enjoy the empty declaration of %lemma_words

my %lemma_words = ();
foreach my $word (lemmatizer(join(" ", keys %bag_of_words))){
    $lemma_words{$word}++;
}

lemmatizer(join(" ", keys %bag_of_words)) is a python function called with Python::Inline and it returns an array. I would like to know if it's possible to declare lemma_words, the hash with lemmatized tokens as keys and occurrences as values within the same line (with a map, an nmap... i don't know. Here is the entire snippet. The only guideline for this college task is that the script must be written in Perl (the less python code is and better it will be)

#!/usr/bin/perl

use strict;
#use warnings;
use open qw/ :std :encoding(UTF-8)/;

use Inline Python => <<'END_OF_PYTHON';

import spacy
from spacy.lang.fr.stop_words import STOP_WORDS as fr_stop
nlp = spacy.load('fr_core_news_md')

def lemmatizer(words):
    doc = nlp(words)
    return list(filter(lambda x: x not in list(fr_stop), list(map(lambda token: token.lemma_ , doc))))

END_OF_PYTHON

open my $fh, "<:encoding(utf-8)", "sortiexml-slurp_3208.xml" or die "$!";
# first level bag_of_words. We remove only punct and num
my %bag_of_words = ();
while (my $ligne = <$fh>){
    next if $ligne !~ /^/;
    # let's extract form of the token (word) from an xml file
    if ($ligne =~ /(.+)<.+/){
        $bag_of_words{$1}++ if length($1) > 2 and $1 !~ /\d+/;
    }
}

close $fh;

# now it's time to lemmatize words and counter new occurrences of each token lemmatized and we have a primitive bag of words
# to apply it after with Cosine Similarity or Naive Bayes for automatic text classification (example)
# we also remove stop words
my %lemma_words = ();
foreach my $word (lemmatizer(join(" ", keys %bag_of_words))){
    $lemma_words{$word}++;
}

# here it's simply a debug print to check errors
for my $key (keys %lemma_words) {
    print "$key => $lemma_words{$key}
";
}

user_4653379 · Answer

Can roll it in a subroutine ... or use the ones provided in libraries

For simple and fast element-frequency counter there is List::MoreUtils::frequency

use List::MoreUtils qw(frequency);

my %freq_count = frequency LIST;

The LIST is any dynamically generated list (that lemmatizer(...) here), or an array variable.

If you'd like a possibility to fine-tune what is counted there is count_by in List::UtilsBy

use List::UtilsBy qw(count_by);

my %freq_count = count_by { $_ } LIST;

Again, LIST is any dynamically produced list or an array variable, and in your case that would be lemmatizer(...) (which returns a list).

The frequency count is returned for values to which the code inside the block evaluates; each element in turn is provided in $_. So with lone $_ the count is for elements themselves.

Both List::MoreUtils and List::UtilsBy probably need be installed from CPAN.

How to declare and assign a hash with count of occurrences in one statement

1 Answers