Should we consider using range [a-z] as a bug?

user_196241 0
Viewed 2996

In my locale (et_EE) [a-z] means:

abcdefghijklmnopqrsšz

So, 6 ASCII chars (tuvwxy) and one from Estonian alphabet (ž) are not included. I see a lot modules which are still using regexes like 

/\A[0-9A-Z_a-z]+\z/

For me it seems wrong way to define range of ASCII alphanumeric chars and i think it should be replaced with:

/\A\p{PosixAlnum}+\z/

Is the first one still considered idiomatic way? Or accepted solution? Or a bug?

Or has last one some caveats?

locale pcre unicode perl regex
5 Answers

If it is exactly what you want then using [a-z] is not wrong.

But it's wrong to believe that English words consist only of [a-zA-Z] or German of [a-zäöüßA-ZÄÖÜ] or names follow [A-Z][a-z]*.

If we want words in any language or writing system (tested against 2,300 languages each 50 K of the most frequent words) we can use something like this:

#!perl

use strict;
use warnings;
use utf8;

use 5.020;    # regex_sets need 5.18

no warnings "experimental::regex_sets";

use Unicode::Normalize;

my $word_frequencies = {};

while (my $line = <>) {
    chomp $line;
    $line = NFC($line);

    # NOTE: will catch "broken" words at end/begin of line
    #       and abbreviations without '.'
    my @words = $line =~ m/(
        (?[ \p{Word} - \p{Digit} + ['`´’] ])
        (?[ \p{Word} - \p{Digit} + ['`´’=⸗‒—-] ])*
    )/xg;
    
    for my $word (@words) {
        $word_frequencies->{$word}++;
    }
}

# now count the frequencies of graphemes the text uses

my $grapheme_frequencies = {};
for my $word (keys %{$word_frequencies}) {
    my @graphemes = m/(\X)/g;
    for my $grapheme (@grapheme) {
        $grapheme_frequencies->{$grapheme} 
            += $word_frequencies->{$word};
    }
}

For a narrower check we can look into the definition of \p{Word}in the Unicode standard https://unicode.org/reports/tr18/#word

word
    \p{alpha}
    \p{gc=Mark}
    \p{digit}
    \p{gc=Connector_Punctuation}
    \p{Join_Control}

Based on \p{Word} we can now define a Regex for e.g. words in the Latin script:

# word:
    \p{Latin}    # \p{alpha}
    \p{gc=Mark}
    # \p{digit}  # we don't want numerals in words
    \p{gc=Connector_Punctuation}
    \p{Join_Control}
user_36318560

for awk, perhaps forcing octal codes on the alphabets should circumvent the inconsistency in awk/poxix/locales

something like

   /[\060-\071       # 0-9
     \101-\132       # A-Z
     \141-\172]/     # a-z

if you want to make them into string constants, perhaps double-the backslashes to ensure the parser/regex engine doesn't get too smart and pre-convert "\101" into A, and give an opening for it to "respect" locale-settings that might not be what you wanted.

"\\101"

user_146721140
Related