Can you extract indexes of data over a threshold from numpy array or pandas dataframe

user_1262602 0
Viewed 348

I am using the following to compare several strings to each other. It's the fastest method I've been able to devise, but it results in a very large 2D array. which I can look at and see what I want. Ideally, I would like to set a threshold and pull the index(es) for each value over that number. To make matters more complicated, I don't want the index comparing the string to itself, and it's possible the string might be duplicated elsewhere so I would want to know if that's the case, so I can't just ignore 1's.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

texts = sql.get_corpus()

vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(texts)
similarity = cosine_similarity(vectors)

sql.get_corups() returns a list of strings, currently 1600ish strings.

Is what I want possible? I've tried using comparing each of the 1.4M combinations to each other using Levenshtein, which works, but it takes 2.5 hours vs half above. I've also tried vecotrs with spacy, which takes days.

scikit-learn spacy levenshtein-distance numpy python
1 Answers

I'm not entirely sure I read your post correctly, but I believe this should get you started:

import numpy as np

# randomly distributed data we want to filter
data = np.random.rand(5, 5)

# get index of all values above a threshold
threshold = 0.5
above_threshold = data > threshold

# I am assuming your matrix has all string comparisons to
# itself on the diagonal
not_ident = np.identity(5) == 0.

# [edit: to prevent duplicate comparisons, use this instead of not_ident]
#upper_only = np.triu(np.ones((5,5)) - np.identity(5))

# 2D array, True when criteria met
result = above_threshold * not_ident
print(result)

# original shape, but 0 in place of all values not matching above criteria
values_orig_shape = data * result
print(values_orig_shape)

# all values that meet criteria, as a 1D array
values = data[result]
print(values)

# indices of all values that meet criteria (in same order as values array)
indices = [index for index,value in np.ndenumerate(result) if value]
print(indices)
user_83172370
Related