Identifying misclassified raw data in after machine learning in Python

user_7775099 0
Viewed 37

I have a pretty basic question. My X data is df['input'], Y data is df['label']. This is my code:

from sklearn.feature_extraction.text import TfidfVectorizer
Xfeatures = df['input']
y = df['label']

tfidf_vec = TfidfVectorizer(max_features= MF,
                           max_df = MAXDF)
X = tfidf_vec.fit_transform(Xfeatures)
featurenames = tfidf_vec.get_feature_names()
X.todense()
df_vec = pd.DataFrame(X.todense(),columns=tfidf_vec.get_feature_names())
df_vec.T
from sklearn.model_selection import train_test_split

x_train,x_test,y_train,y_test = train_test_split(X, y,test_size = 0.33,random_state = 28)

This is the model that I run for text classification:

from sklearn.svm import SVC
lr_model = SVC()
lr_model.fit(x_train,y_train)
y_pred = lr_model.predict(x_test)
# Acccuracy
print(classification_report(y_test,y_pred))
print(accuracy_score(y_test,y_pred))

I would like to identify the those that are misclassified (i.e. df['input']). I can write down predicted and actual the categories into csv, but not the text that is misclassified (or training data in general):

import csv
rows = zip(y_test, y_pred)
with open(r"C:\Users\erdem\Desktop\data.csv", "w", newline="") as f:
    writer = csv.writer(f)
    for row in rows:
        writer.writerow(row)
machine-learning python
1 Answers

Try to go with

X[y_test != y_pred]

y_test != y_pred will be a binary array with True on misclassified data (and False on correct predictions): you can use that as indices for your X (or Xfeatures).

user_179020180
Related