I have a problem concerning the tfidfVectorizer. My problem is that I have 3 columns, one is the text that needs to be vectorized and the two others are already numbers, so I only need to vectorize one of them. I have read that you need to vectorize your data after you have split it into training and test set, so I have split my data set like so:
X = df[['cleaned_tweet_text', 'polarity', 'subjectivity']] # The Tweets
y = df['cyberbullying_type'] # The Label
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, random_state = 42)It is the cleaned_tweet_text that needs to be vectorized
I have tried this(see below) but I am not sure this is the right way.
tfidf = TfidfVectorizer(max_features = 1000)
X_train_tfidf = tfidf.fit_transform(X_train.cleaned_tweet_text)
X_test_tfidf = tfidf.transform(X_test.cleaned_tweet_text)It does not give me an error, and if I print out X_train_tfidf I get this:
(0, 217) 0.41700972853730645
(0, 118) 0.16283369998713235
(0, 758) 0.16948694862672925
(0, 404) 0.20143376247898365
(0, 626) 0.4426572817169202
(0, 356) 0.20217167680038242
(0, 871) 0.4634256150008882
(0, 65) 0.3606189681792524
(0, 565) 0.38556256201243433
(1, 719) 0.29478675756557454
(1, 919) 0.30596230567496185
(1, 698) 0.36538974359723864
(1, 485) 0.816429056367109
(1, 118) 0.13936199719971182
(2, 342) 0.17134974750083107
(2, 256) 0.18449190025596335
(2, 110) 0.3604602574432005
(2, 290) 0.39210201833562014
(2, 648) 0.3538174461369334
(2, 161) 0.2742199778844052
(2, 251) 0.3864257748655211
(2, 128) 0.26063790594719993
(2, 599) 0.18251158997125277
(2, 123) 0.39339155686431243
(2, 360) 0.21729849596293152does that mean it works? so now I can put it into a classifier?