Clustering script fails with German, but works like expected with English

user_1992004 0
Viewed 242

I have a script to cluster keywords, utilizing pandas and polyfuzz. With English, it works like expected. Trying to use the script with keywords in German, the script recognizes multiple keywords wrongly.

What means "wrongly recognized": clustering recognizes the first and second word in the keyword. And as you can see on the screenshot, columns G and H (First Word and Second Word) contain other words, then corresponding keywords in column B (Keyword):

enter image description here

The script fails not always with German - multiple keywords are clustered correctly. But the part of wrongly recognized keywords is very high, up to 20%.

Could somebody explain to me why the script failed with German keywords and, in the best case, improve the script enabling it to work with German?

Here is the part of the script, which does clustering:

# find keywords from one column in another in any order and count the frequency
df_matched['Cluster Name'] = df_matched['Cluster Name'].str.strip()
df_matched['Keyword'] = df_matched['Keyword'].str.strip()

df_matched['First Word'] = df_matched['Cluster Name'].str.split(" ").str[0] 
df_matched['Second Word'] = df_matched['Cluster Name'].str.split(" ").str[1] 
df_matched['Total Keywords'] = df_matched['First Word'].str.count(' ') + 1

def ismatch(s):
    A = set(s["First Word"].split())
    B = set(s['Keyword'].split())
    return A.intersection(B) == A

df_matched['Found'] = df_matched.apply(ismatch, axis=1)

df_matched = df_matched. fillna('')

def ismatch(s):
    A = set(s["Second Word"].split())
    B = set(s['Keyword'].split())
    return A.intersection(B) == A
df_matched['Found 2'] = df_matched.apply(ismatch, axis=1)


# todo - document this algo. Essentially if it matches on the second word only, it renames the cluster to the second word
# clean up code nd variable names
df_matched.loc[(df_matched["Found"] == False) & (df_matched["Found 2"] == True), "Cluster Name"] = df_matched["Second Word"]
df_matched.loc[(df_matched["Found"] == False) & (df_matched["Found 2"] == False), "Cluster Name"] = "zzz_no_cluster_available"

# count cluster_size
df_matched['Cluster Size'] = df_matched['Cluster Name'].map(df_matched.groupby('Cluster Name')['Cluster Name'].count())
df_matched.loc[df_matched["Cluster Size"] == 1, "Cluster Name"] = "zzz_no_cluster_available"


df_matched = df_matched.sort_values(by="Cluster Name", ascending=True)

Here are two datasets:

And here, the working Colab with the whole script.

pandas python
1 Answers

I opened the full code to understand where df_matched came from. I'm not 100% sure of what you are trying to do, but I think that the problem comes from before the snippet you shared here.

It comes from the way that df_matched is created. It uses fuzzy matching to create clusters. So the words of "Cluster Name" are not all guaranteed to be present in "Keyword".

If you run the code for the English data, and check the words in position -1 and -2 (last two words of the Cluster Name) instead of 0 and 1...

df_matched['First Word'] = df_matched['Cluster Name'].str.split(" ").str[-1] 
df_matched['Second Word'] = df_matched['Cluster Name'].str.split(" ").str[-2]

...then calculate how many of them are not found...

print((~df_matched["Found"]).sum())
print((~df_matched["Found 2"]).sum())
# 140
# 10

...you can see that for 104 out of 158 rows, the last word is not part of the keywords.

(I don't know if you care about the first two words more than the last two... but this looks worse than the 20% you noticed in the German data.)

For the German one the problem is more visible because this language uses a lot of compound words and many frequent suffixes (e.g., "ung")... So they will fuzzy-match a lot.

Example of df_matched for german: the "From words are not present in "To"... but there are large overlaps

Example of df_matched for German: the "From" words are not present in "To"... but there are large overlaps.

English example

This is df_matched for English: some words of "From" are not even close to the words in "To"... and similarity score can be worse than in the German dataset.

Possible improvements

I think that the part where you could improve the clustering is this (from the colab notebook)

df_1_list = df_1.Keyword.tolist()  # create list from df
model = PolyFuzz("TF-IDF")

cluster_tags = df_1_list[::]
cluster_tags = set(cluster_tags)
cluster_tags = list(cluster_tags)

print("Cleaning up the cluster tags.. Please be patient!")
substrings = {w1 for w1 in tqdm(cluster_tags) for w2 in cluster_tags if w1 in w2 and w1 != w2}
longest_word = set(cluster_tags) - substrings
longest_word = list(longest_word)
shortest_word_list = list(set(cluster_tags) - set(longest_word))

try:
    model.match(df_1_list, shortest_word_list)
except ValueError:
    print("Empty Dataframe, Can't Match - Check the URL Filter!")
    sys.exit()

model.group(link_min_similarity=sim_match_percent)
df_matched = model.get_matches()

Here you compute the similarity between df_1_list and shortest_word_list. shortest_word_list is created by looking for substrings, which might lead to weird clusters is German because of compound words.

You could try and normalize the text with (language-specific) ​stemming or lemmatization before / instead of checking for substrings and creating clusters. This should help and transform each word in their "root form" and retain their meaning.

Yoy can use the spaCy library, which provide language-specific pretrained models for stemming, embedding and other language operations. You can select the correct model for each language and use the lemmatization function to replace each word of df_1_list with their "base form" before trying to cluster.

Lemmatization example

import spacy

nlp = spacy.load("en_core_web_sm") # load English or German model
lemmatizer = nlp.get_pipe("lemmatizer")
print(lemmatizer.mode)  # 'rule'

doc = nlp("I was reading the paper.")
print([token.lemma_ for token in doc])
# ['I', 'be', 'read', 'the', 'paper', '.']

Link to spaCy German model: https://spacy.io/models/de

user_181302890
Related