How to tokenize compound words?

user_4538768 0
Viewed 259

Having an original list element such as ["southnorth"], I'd like to add a space based on the list ["south", "north", "island"]. Then, the list would be changed from ['southnorth'] to ['south','north'] as long as the list which we base the tokenization contains ['south', 'north'].

However, if there is a list ["south", "island"] then the list ["southnorth"] should be kept together as it is.

I thought in something as follow:

list1= ['southnorth']
#list2= ['south','north','island']
list2=['south','island']

str1= " ".join(list1)
str2= " ".join(list2)

Get the alternators to apply regex:
list_compound = sorted(list1 + list2, key=len)
alternators = '|'.join(map(re.escape, list_compound)
regex = re.compile(r''.format(alternators)

str1_split = re.sub(r'({})'.format(alternators),r'\1 ',str1,0, re.IGNORECASE)

str2_split = re.sub(r'({})'.format(alternators),r'\1 ',str2,0, re.IGNORECASE)

However, the above is failing because I need to ensure the order of the sequences. For instance, to decompose ["southnorth"] I need to ensure the other list has ["south", "north"]. Otherwise, keep it in the original form.

tokenize python-3.x python regex
1 Answers

Not the prettiest solution and probably not the best performing, but here is a trivial brute force attempt:

def tokenize(word, tokens):
    tokenized_word = word
    for t in tokens:
        tokenized_word = tokenized_word.replace(t, f"{t} ").strip()

    for w in tokenized_word.split(" "):
        if w.strip() not in tokens:
            return word

    return tokenized_word


tokens = ["south", "north", "island"]

assert tokenize("south", tokens) == "south"
assert tokenize("southnorth", tokens) == "south north"
assert tokenize("islandsouthnorth", tokens) == "island south north"
assert tokenize("southwestnorth", tokens) == "southwestnorth"
user_20111470
Related