NLTK - Python : How to format a raw text

user_8925703 0
Viewed 650

Do you know if with NLTK (or any other NLP) & Python I can format a raw text (no punctuation, nor capitals nor linebreaks between paragraphs)?

I've gone through the documentation but I can't find anything that would help me with this task.

Example:

Input:

python is an interpreted high-level general-purpose programming language created by guido van rossum and first released in 1991 python has a design philosophy that emphasizes code readability notably using significant whitespace it provides constructs that enable clear programming on both small and large scales in July 2018, van rossum stepped down as the leader in the language community

Output:

Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python has a design philosophy that emphasizes code readability, notably using significant whitespace. It provides constructs that enable clear programming on both small and large scales. In July 2018, Van Rossum stepped down as the leader in the language community.

Thank you,

nltk python
1 Answers

Interesting question. As for the inserting of boundaries, you can train NLTK's tokenizer (or sentence splitter) (plenty of docs on that if you google). One thing you can try is to get some text that's sentence-splitted, remove punctuation and then train and see what you get. Something like the following (below). As indicated already, the algorithm probably relies quite heavily on punctuation, and in any case the code below doesn't work for your example sentence, but perhaps if you use some other/larger/different domain training text, it could be worth trying out. Not entirely sure if this would also work for inserting comma's and other (non-sentence-final/initial) punctuation.

from nltk.corpus import gutenberg
from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktTrainer
import re

text = ""
for file_id in gutenberg.fileids():
    text += gutenberg.raw(file_id)
# remove punctuation
text = re.sub('[\.\?!]\n', '\n', text) #  you will probably want to include some other potential sentence final punctuation here
trainer = PunktTrainer()
trainer.INCLUDE_ALL_COLLOCS = True
trainer.train(text)
tokenizer = PunktSentenceTokenizer(trainer.get_params())
sentences = "python is an interpreted high-level general-purpose programming language created by guido van rossum and first released in 1991 python has a design philosophy that emphasizes code readability notably using significant whitespace it provides constructs that enable clear programming on both small and large scales in July 2018, van rossum stepped down as the leader in the language community"
 print(tokenizer.tokenize(sentences))
user_43375220
Related