if I have this code:
source_text = 'To convert some docs, <g id="1">just click “Add Books” </g>button and then click “<g id="2">Convert”.</g> Set the output format and click <g id="3">“OK”</g>.'
abbreviations = ['e.g', 'i.e']
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
tokenizer._params.abbrev_types.update(abbreviations)
tokenizer_dicts = tokenizer.tokenize(source_text)
for tokenized in tokenizer_dicts:
print(tokenized)
this will output:
To convert some docs, <g id="1">just click “Add Books” </g>button and then click “<g id="2">Convert”.
</g> Set the output format and click <g id="3">“OK”</g>.
whereas the expected result is:
To convert some docs, <g id="1">just click “Add Books” </g>button and then click “<g id="2">Convert”.</g>
Set the output format and click <g id="3">“OK”</g>.
Notice the </g> at the start of the second string is now "moved and appended" to the first string
How do I give it a pattern so it splits like the expected result? or any way around it?