Regex to capture multilines from specific expression until first blank line in Perl

Question

i have this file and i would like to match everything between #sent_id=\d+ and blank line. Here is my file :

# newdoc
# newpar
# sent_id = 1
# text = 2019, un bon cru pour les amateurs de technologie.
1   2019    2019    NUM _   _   5   nummod  _   SpaceAfter=No
2   ,   ,   PUNCT   _   _   1   punct   _   _
3   un  un  DET _   Definite=Ind|Gender=Masc|Number=Sing|PronType=Art   5   det _   _
4   bon bon ADJ _   Gender=Masc|Number=Sing 5   amod    _   _
5   cru cru NOUN    _   Gender=Masc|Number=Sing 0   root    _   _
6   pour    pour    ADP _   _   8   case    _   _
7   les le  DET _   Definite=Def|Gender=Masc|Number=Plur|PronType=Art   8   det _   _
8   amateurs    amateur NOUN    _   Gender=Masc|Number=Plur 5   nmod    _   _
9   de  de  ADP _   _   10  case    _   _
10  technologie technologie NOUN    _   Gender=Fem|Number=Sing  8   nmod    _   SpaceAfter=No
11  .   .   PUNCT   _   _   5   punct   _   SpacesAfter=


# sent_id = 2
# text = L’année a été généreuse en innovations prometteuses. Voici les six tendances qui pourraient bousculer notre paysage technologique.
1   L’  L’  PROPN   _   _   5   nsubj   _   SpaceAfter=No
2   année   année   NOUN    _   Gender=Fem|Number=Sing  1   nmod    _   _
3   a   avoir   AUX _   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   5   aux:tense   _   _
4   été être    AUX _   Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part    5   cop _   _
5   généreuse   généreux    ADJ _   Gender=Fem|Number=Sing  0   root    _   _
6   en  en  ADP _   _   7   case    _   _
7   innovations innovation  NOUN    _   Gender=Fem|Number=Plur  5   obl _   _
8   prometteuses    prometteur  ADJ _   Gender=Fem|Number=Plur  7   amod    _   SpaceAfter=No
9   .   .   PUNCT   _   _   10  punct   _   _
10  Voici   voici   VERB    _   _   5   parataxis   _   _
11  les le  DET _   Definite=Def|Gender=Fem|Number=Plur|PronType=Art    13  det _   _
12  six six NUM _   _   13  nummod  _   _
13  tendances   tendance    NOUN    _   Gender=Fem|Number=Plur  10  obj _   _
14  qui qui PRON    _   PronType=Rel    15  nsubj   _   _
15  pourraient  pouvoir VERB    _   Mood=Cnd|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   13  acl:relcl   _   _
16  bousculer   bousculer   VERB    _   VerbForm=Inf    15  xcomp   _   _
17  notre   son DET _   Gender=Masc|Number=Sing|Poss=Yes|PronType=Prs   18  det _   _
18  paysage paysage NOUN    _   Gender=Masc|Number=Sing 16  obj _   _
19  technologique   technologique   ADJ _   Gender=Masc|Number=Sing 18  amod    _   SpaceAfter=No
20  .   .   PUNCT   _   _   5   punct   _   SpacesAfter=


# sent_id = 3
# text = A Hongkong, des centaines d’arrestations lors d’affrontements, une situation qui semble « désespérée ».
1   A   à   ADP _   _   2   case    _   _
2   Hongkong    Hongkong    PROPN   _   _   0   root    _   SpaceAfter=No
3   ,   ,   PUNCT   _   _   2   punct   _   _
4-5 des _   _   _   _   _   _   _   _
4   de  de  ADP _   _   6   case    _   _
5   les le  DET _   Definite=Def|Gender=Fem|Number=Plur|PronType=Art    6   det _   _
6   centaines   centaine    NOUN    _   Gender=Fem|Number=Plur  2   nmod    _   _
7   d’  d’  PROPN   _   _   6   appos   _   SpaceAfter=No
8   arrestations    arrestation NOUN    _   Gender=Fem|Number=Plur  7   flat:name   _   _
9   lors    lors    ADV _   _   10  case    _   _
10  d’  d’  PROPN   _   _   6   nmod    _   SpaceAfter=No
11  affrontements   affrontement    NOUN    _   Gender=Masc|Number=Plur 10  flat:name   _   SpaceAfter=No
12  ,   ,   PUNCT   _   _   14  punct   _   _
13  une un  DET _   Definite=Ind|Gender=Fem|Number=Sing|PronType=Art    14  det _   _
14  situation   situation   NOUN    _   Gender=Fem|Number=Sing  10  appos   _   _
15  qui qui PRON    _   PronType=Rel    16  nsubj   _   _
16  semble  sembler VERB    _   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   14  acl:relcl   _   _
17  «   «   PUNCT   _   _   18  punct   _   SpacesAfter= 
18  désespérée  désespéré   ADJ _   Gender=Fem|Number=Sing  16  xcomp   _   SpacesAfter= 
19  »   »   PUNCT   _   _   18  punct   _   SpaceAfter=No
20  .   .   PUNCT   _   _   10  punct   _   SpacesAfter=


# sent_id = 4
# text = Des dizaines de milliers de personnes ont défilé dans la mégapole. « C’est triste que nos revendications de 2019 doivent être reportées à 2020 », a souligné un des organisateurs.
1-2 Des _   _   _   _   _   _   _   _
1   De  de  ADP _   _   3   case    _   _
2   les le  DET _   Definite=Def|Gender=Fem|Number=Plur|PronType=Art    3   det _   _
3   dizaines    dizaine NOUN    _   Gender=Fem|Number=Plur  9   nsubj   _   _
4   de  de  ADP _   _   5   case    _   _
5   milliers    millier NOUN    _   Gender=Masc|Number=Plur 3   nmod    _   _
6   de  de  ADP _   _   7   case    _   _
7   personnes   personne    NOUN    _   Gender=Fem|Number=Plur  5   nmod    _   _
8   ont avoir   AUX _   Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   9   aux:tense   _   _
9   défilé  défiler VERB    _   Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part    31  ccomp   _   _
10  dans    dans    ADP _   _   12  case    _   _
11  la  le  DET _   Definite=Def|Gender=Fem|Number=Sing|PronType=Art    12  det _   _
12  mégapole    mégapole    NOUN    _   Gender=Fem|Number=Sing  9   obl:arg _   SpaceAfter=No
13  .   .   PUNCT   _   _   17  punct   _   _
14  «   «   PUNCT   _   _   17  punct   _   SpacesAfter= 
15  C’  C’  PROPN   _   _   17  nsubj   _   SpaceAfter=No
16  est être    AUX _   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   17  cop _   _
17  triste  triste  ADJ _   Gender=Masc|Number=Sing 9   xcomp   _   _
18  que que SCONJ   _   _   23  mark    _   _
19  nos son DET _   Gender=Fem|Number=Plur|Poss=Yes|PronType=Prs    20  det _   _
20  revendications  revendication   NOUN    _   Gender=Fem|Number=Plur  23  nsubj   _   _
21  de  de  ADP _   _   22  case    _   _
22  2019    2019    NUM _   _   20  nmod    _   _
23  doivent devoir  VERB    _   Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin   17  ccomp   _   _
24  être    être    AUX _   VerbForm=Inf    25  aux:pass    _   _
25  reportées   reporter    VERB    _   Gender=Fem|Number=Plur|Tense=Past|VerbForm=Part 23  xcomp   _   _
26  à   à   ADP _   _   27  case    _   _
27  2020    2020    NUM _   _   25  obl:arg _   SpacesAfter= 
28  »   »   PUNCT   _   _   17  punct   _   SpaceAfter=No
29  ,   ,   PUNCT   _   _   9   punct   _   _
30  a   avoir   AUX _   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   31  aux:tense   _   _
31  souligné    souligner   VERB    _   Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part    0   root    _   _
32  un  un  PRON    _   Gender=Masc|Number=Sing 31  nsubj   _   _
33-34   des _   _   _   _   _   _   _   _
33  de  de  ADP _   _   35  case    _   _
34  les le  DET _   Definite=Def|Gender=Masc|Number=Plur|PronType=Art   35  det _   _
35  organisateurs   organisateur    NOUN    _   Gender=Masc|Number=Plur 32  nmod    _   SpaceAfter=No
36  .   .   PUNCT   _   _   31  punct   _   SpacesAfter=

I would like to capture everything between #sent_id and a new #sent_id.

I've tried this but it matches all the file.

sub extract_dependancies{

    open my $fh, "<:encoding(utf-8)", shift or die "$!
";
    undef $/;
    my $file = <$fh>;
   # sent_id to blank line
    while($file =~ /^#\ssent.+\d+(.+)^$/msg){
        my $sentence = $1;
        #next if $sentence !~ /d+	obj/;
    }
}

The scalar sentence retrieve all the file. I would like to have sentence by sentence. sentence should be contain the content between two # sent_id like this.

# text = 2019, un bon cru pour les amateurs de technologie.
1   2019    2019    NUM _   _   5   nummod  _   SpaceAfter=No
2   ,   ,   PUNCT   _   _   1   punct   _   _
3   un  un  DET _   Definite=Ind|Gender=Masc|Number=Sing|PronType=Art   5   det _   _
4   bon bon ADJ _   Gender=Masc|Number=Sing 5   amod    _   _
5   cru cru NOUN    _   Gender=Masc|Number=Sing 0   root    _   _
6   pour    pour    ADP _   _   8   case    _   _
7   les le  DET _   Definite=Def|Gender=Masc|Number=Plur|PronType=Art   8   det _   _
8   amateurs    amateur NOUN    _   Gender=Masc|Number=Plur 5   nmod    _   _
9   de  de  ADP _   _   10  case    _   _
10  technologie technologie NOUN    _   Gender=Fem|Number=Sing  8   nmod    _   SpaceAfter=No
11  .   .   PUNCT   _   _   5   punct   _   SpacesAfter=

for the first sentence and so forth. But either I capture all the file or nothing at all.

user_4653379 · Answer

One way: read the file in chunks of text between empty lines.

use warnings;
use strict;
use feature 'say';

die "Usage: $0 file
" if not @ARGV;

my @sentences;

# First pass lines until the first  "# sent_id =..." (and that one)
while (<>) { last if /^\s*#\s*sent_id/ }

local $/ = "

";  # read in blocks of up to an empty line

while (<>) {
    # Remove the leading line "# sent_id =...", return the rest
    push @sentences, s{ ^ \#\s* sent_id\s* = \s*[0-9]+ .*?
 }{}xr;
}

say for @sentences;

The first while <> is used to get to and pass the first sent_id line. Then the next while (<>) starts from where the first one stopped, and reads the file by blocks delimited by an empty line.

The first line in each block, with sent_id, is matched and removed, while the rest is returned (by the "non-destructive" /r modifier) and added to the array.

This retains the trailing empty line in each "sentence," easily removed if need be.

I read the file using the magical "null filehandle" <> and do everything in the main program for simplicity, but please do put it in a nice subroutine as in the question. Then it can return a reference to @sentences.

Note the local, which in a short program like above doesn't matter but is crucial to use in general; with it the global variable on which it is applied (here $/) is restored upon leaving the scope. The code in the question changes the global $/ variable for all code in the interpreter.

Or use regex to parse the file, as asked for and attempted.

One way: iterate using the # sent_id = ... lines as token

use warnings;
use strict; 
use feature 'say';

die "Usage: $0 file
" if not @ARGV;

my $content = do { local $/; <> };

my @sentences;

while ( $content =~ / \#\s* sent_id \s*=\s* [0-9]+ .*? 
 /gx ) {
    push @sentences, $content =~ /\G (.*?) (?:

|$)/sx;
    #say "pos = ", pos $content;
}

say for @sentences;

The \G anchors to where the previous match with /g modifier left off. So the pattern with it (in the push... line) starts matching from after what the pattern in the while condition matched, so it matches everything from after the previous # sent_id... line to the next empty line.

Please see docs for how the combination of /g and \G works and can be used, for example in "Global Matching" in perlretut

Or, get everything in one shot

use warnings;
use strict;
use feature 'say';

die "Usage: $0 file
" if not @ARGV;

my $content = do { local $/; <> };

my @sentences = 
    $content =~ / \#\s*sent_id\s*=\s*[0-9]+ .*?
 (.+?) (?:

|$) /gsx;

say for @sentences;

This is in my opinion the trickiest and least robust solution.

user_15914583 · Answer

perl -ne '$f=0 if /sent_id/; print "$_" if $f; $f=1 if /sent_id/' french.in > french.out

This one-liner captures everything between one #sent_id and the next #sent_id.

How does it work:

perl -MO=Deparse -ne '$f=0 if /sent_id/; print "$_" if $f; $f=1 if /sent_id/' french.in

LINE: while (defined($_ = readline ARGV)) {
    $f = 0 if /sent_id/;
    print "$_" if $f;
    $f = 1 if /sent_id/;
}

While the first line is being processed, the tag #sent_id = 1 is matched and switches $f into the false state: $f=0 if /sent_id/;

The next statement, print "$_" if $f is therefore not executed.

Then, while the first line is still being processed, the tag #sent_id = 1 is matched again and switches $f into the true state: $f=1 if /sent_id/;

The while loop enters the next iteration and moves to the next line: "# text = 2019, un bon cru pour les amateurs de technologie."

Since $f is now in the true state, this line is printed and so the following lines, up to the line which starts with the tag # sent_id = 2.

Here the $f switches to the false state again: $f = 0 if /sent_id/; and the next statement, print "$_" if $f is not executed. While still on the same line, the tag # sent_id = 2 is matched again and turns $f into the true state: $f = 1 if /sent_id/;

Now the while loop moves to the next line: "# text = L’année a été généreuse en innovations prometteuses. ..." Since $f is now in the true state, this line and the following ones (up the tag # sent_id = 3) are printed.

user_12313309 · Answer

Please investigate the following solution for compliance with your task

define local end of the record on empty line
read records from <> in map loop
utilize regex to capture only what you interested in
store result in @blocks array
show @blocks elements with say (optional)

NOTE: run as ./script.pl datafile

use strict;
use warnings;
use feature 'say';

my @blocks;

{
    local $/ = "

";
    @blocks = map { /(# sent_id = .*)/s } <>;
}

say for @blocks;

Regex to capture multilines from specific expression until first blank line in Perl

3 Answers