How to predict after training data using naive bayes with python?

user_7967778 0
Viewed 9009

I have got a dataset which contains just two useful columns for training my model, first is news heading and the second is category of news.

So, I got the following training command running successfully using python:

import re
import numpy as np
import pandas as pd
# the Naive Bayes model
from sklearn.naive_bayes import MultinomialNB
# function to split the data for cross-validation
from sklearn.model_selection import train_test_split
# function for transforming documents into counts
from sklearn.feature_extraction.text import CountVectorizer
# function for encoding categories
from sklearn.preprocessing import LabelEncoder


# grab the data
news = pd.read_csv("/Users/helloworld/Downloads/NewsAggregatorDataset/newsCorpora.csv",encoding='latin-1')
news.head()

def normalize_text(s):
    s = s.lower()

    # remove punctuation that is not word-internal (e.g., hyphens, apostrophes)
    s = re.sub('\s\W',' ',s)
    s = re.sub('\W\s',' ',s)

    # make sure we didn't introduce any double spaces
    s = re.sub('\s+',' ',s)

    return s

news['TEXT'] = [normalize_text(s) for s in news['TITLE']]

# pull the data into vectors
vectorizer = CountVectorizer()
x = vectorizer.fit_transform(news['TEXT'])

encoder = LabelEncoder()
y = encoder.fit_transform(news['CATEGORY'])

# split into train and test sets
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)

nb = MultinomialNB()
nb.fit(x_train, y_train)

So my question is, how can I give a new set of data (e.g. Just news heading) and tell the program to predict the news category using python sklearn command?

P.S. My training data is like:

enter image description here

scikit-learn naivebayes python
2 Answers

You should train the model using the training data (as you did) and then you should predict using new data (the test data).

Do the following:

nb = MultinomialNB()
nb.fit(x_train, y_train)

y_predicted = nb.predict(x_test)

Now, if you want to evaluate the predictions based on the **accuracy you can do the following:**

from sklearn.metrics import accuracy_score

accuracy_score(y_test, y_predicted)

Similarly, you can calculate other metrics.

Finally, we can see all the available metrics here !

EDIT 1

When you type:

 y_predicted = nb.predict(x_test)

y_predicted will contain numerical values that correspond to your categories.

To project back these values and get the labels you can do:

y_predicted_labels = encoder.inverse_transform(y_predicted)
user_50250090
user_50557880
Related