How to use regex to scrape entire sentences from fragments in python

user_14975348 0
Viewed 167

I have a vtt file as following

WEBVTT

1
00:00:05.210 --> 00:00:07.710
In this lecture, we're
going to talk about

2
00:00:07.710 --> 00:00:10.815
pattern matching in strings
using regular expressions.

3
00:00:10.815 --> 00:00:13.139
Regular expressions or regexes

4
00:00:13.139 --> 00:00:15.825
are written in a condensed
formatting language.

I want to extract the fragments from the file and merge them into sentences. The output should look something like this

['In this lecture, we're going to talk about pattern matching in strings using regular expressions.', 'Regular expressions or regexes are written in a condensed formatting language.'

I am able to extract the fragments using this

pattern = r"[A-z0-9 ,.*?='\";\n-\/%$#@!()]+"

content = [i for i in re.findall(pattern, text) if (re.search('[a-zA-Z]', i))]

I am not sure how to extract entire sentences instead of fragments.

Also note that this is just a sample of the vtt file. The entire vtt file contains around 630 fragments and some of the fragments also contains integers and other special characters in them

Any help is appreciated

python regex
3 Answers

Using re.sub we can try first removing the unwanted repetitive text. Then, do a second replacement to replace remaining newlines with single spaces:

inp = """1
00:00:05.210 --> 00:00:07.710
In this lecture, we're
going to talk about

2
00:00:07.710 --> 00:00:10.815
pattern matching in strings
using regular expressions.

3
00:00:10.815 --> 00:00:13.139
Regular expressions or regexes

4
00:00:13.139 --> 00:00:15.825
are written in a condensed
formatting language."""

output = re.sub(r'(?:^|\r?\n)\d+\r?\n\d{2}:\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}:\d{2}\.\d{3}\r?\n', '', inp)
output = re.sub(r'\r?\n', ' ', output)
sentences = re.findall(r'(.*?\.)\s*', output)
print(sentences)

This prints:

["In this lecture, we're going to talk about pattern matching in strings using regular expressions.",
 'Regular expressions or regexes are written in a condensed formatting language.']
user_18632290

I find @Tim Biegeleisen's solution with the complex regex and multiple substitutions a bit confusing so here's another option.

import re

_file = """1
00:00:05.210 --> 00:00:07.710
In this lecture, we're
going to talk about

2
00:00:07.710 --> 00:00:10.815
pattern matching in strings
using regular expressions.

3
00:00:10.815 --> 00:00:13.139
Regular expressions or regexes

4
00:00:13.139 --> 00:00:15.825
are written in a condensed
formatting language.
"""

non_fragments = re.compile(r'$|\d+($|:\d+.* --> \d+.*$)')

full_text = " ".join([line for line in _file.splitlines() if not non_fragments.match(line)])
sentences = full_text.split('. ')

This returns:

print(full_text)
In this lecture, we're going to talk about pattern matching in strings using regular expressions. Regular expressions or regexes are written in a condensed formatting language.

print(sentences)
["In this lecture, we're going to talk about pattern matching in strings using regular expressions", 'Regular expressions or regexes are written in a condensed formatting language.']

And as an extra (small) bonus this option is at least twice as fast as the one using re.sub/re.findall

Most efficient when pre-compiling the regex. Didn't test with a very large sample.

%%timeit
_full_text = " ".join([line for line in _file.splitlines() if not non_fragments.match(line)])
_sentences = _full_text.split('. ')
6.75 µs ± 831 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

But it's still faster if we include re.compile processing for each iteration

%%timeit
non_fragments = re.compile(r'$|\d+($|:\d+.* --> \d+.*$)')
_full_text = " ".join([line for line in _file.splitlines() if not non_fragments.match(line)])
_sentences = _full_text.split('. ')  
7.97 µs ± 1.13 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

This one at least twice as long. Not sure how this behaves with a very large text.

%%timeit
output = re.sub(r'(?:^|\r?\n)\d+\r?\n\d{2}:\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}:\d{2}\.\d{3}\r?\n', '', _file)
output = re.sub(r'\r?\n', ' ', output)
sentences = re.findall(r'(.*?\.)\s*', output)
15.2 µs ± 423 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
user_150675020

You can also match the structure of the data in the file to make sure that it is present.

^\d+\r?\n\d{2}:\d{2}:\d{2}\.\d{3} -->.*\r?\n((?:(?!\d+\r?\n\d\d:).*(?:\r?\n|$))*)

Expanation

  • ^ Start of string
  • \d+\r?\n\d{2}:\d{2}:\d{2}\.\d{3} -->.* Match 1+ digits, a newline and the time like pattern
  • \r?\n match a newline
  • ( Capture group 1
    • (?: Non capture group
      • (?!\d+\r?\n\d\d:).*(?:\r?\n|$) Match the whole line if it does not start with a time like pattern
    • )* Close the group and repeat 0+ times to match all the lines
  • ) Close group 1

See an online regex demo | Python demo

Match all the text after time pattern in a capture group that will be returned by re.findall in a list.

Then join all the parts an empty string, replace the newlines by a space and split on 1 or more whitespace chars after a dot.

Example code

regex = r"^\d+\r?\n\d{2}:\d{2}:\d{2}\.\d{3} -->.*\r?\n((?:(?!\d+\r?\n\d\d:).*(?:\r?\n|$))*)"
content = [i for i in re.split(r"(?<=\.)\s+", re.sub(r"[\r\n]+", " ", "".join(re.findall(regex, text, re.M)))) if i]
print(content)

Output

["In this lecture, we're going to talk about pattern matching in strings using regular expressions.", 'Regular expressions or regexes are written in a condensed formatting language.']
user_54249880
Related