I m trying to tokenize English sentences using the BlingFire package and have come across a peculiar scenario -
The package does not treat punctuations followed by small cases as a sentence boundary. For eg. - With upper case sentences, tokenization identifies boundaries properly -
>>> text = 'This is a test. This is a test! This is a test? This is a test, This is a test;'
>>> print(text_to_sentences(text).strip())
**This is a test.**
**This is a test!**
**This is a test?**
**This is a test, This is a test;**
However, replacing the upper cases with lower case fails and returns the following -
>>> text = 'this is a test. this is a test! this is a test? this is a test, this is a test;'
>>> print(text_to_sentences(text))
**this is a test. this is a test! this is a test? this is a test, this is a test;**
Am I missing something that needs to be passed to the text_to_sentences() function for the lowercase words to be identified or is this just how BlingFire works?