tokenize individual sentences from a string of sentences

user_17178435 0
Viewed 56

I have an assignment question where i have to find a way to tokenize individual sentences from a string of sentence. A sentence is any sequence of words that is terminated by a full stop (and including the full stop itself). If no sentences could be segmented then it returns an empty list. I am also guaranteed that a document will not begin with the full stop character. This is still a basic level of python.

this is the code i have started with using the split function

def sentence_segmentation(document):
    """splits each word"""
    sentence_new = document.split(".")
    final = list(sentence_new)
    return final

however im unsure on how to keep the delimeter and especially when there is more than one in the sentence?

i have included the test cases

Test 1

document = "sent1. sent2. sent3. sent4. sent5."
sentences = sentence_segmentation(document)
print(sentences)

result

['sent1.', 'sent2.', 'sent3.', 'sent4.', 'sent5.']

test 2

document = "sent 1. sent 2... sent 3.... sent 4. sent 5.."
sentences = sentence_segmentation(document)
print(sentences)

result

['sent 1.', 'sent 2...', 'sent 3....', 'sent 4.', 'sent 5..']

test 3

document = "sent1.sent2.sent3.sent4.sent5."
sentences = sentence_segmentation(document)
print(sentences)

result

['sent1.sent2.sent3.sent4.sent5.']

Thanks !

tokenize python split string
1 Answers

It looks like you want to split with one or more whitespaces that are preceded with a . char.

Then you can use

import re
documents = ["sent1. sent2. sent3. sent4. sent5.", "sent 1. sent 2... sent 3.... sent 4. sent 5..", "sent1.sent2.sent3.sent4.sent5."]
dot_space_regex = re.compile(r'(?<=\.)\s+')
for doc in documents:
    print(dot_space_regex.split(doc))

See the online demo. Output:

['sent1.', 'sent2.', 'sent3.', 'sent4.', 'sent5.']
['sent 1.', 'sent 2...', 'sent 3....', 'sent 4.', 'sent 5..']
['sent1.sent2.sent3.sent4.sent5.']

See the regex demo.

  • (?<=\.) - a positive lookbehind that matches a location that is immediately preceded with a . char
  • \s+ - one or more whitespaces.
user_38329700
Related