cross_val_score and StratifiedKFold give different result

The reason for better results in the 2nd case is that the train_cv dataset has already been fitted and transformed by the count_vectorizer.

In the former case, within each CV fold, you fit the vectorizer on the training data and transform the validation data. This is the correct approach because the vectorizer does not see the validation data during fit.

In order to do the same with the cross_val_score(), you should create a pipeline that contains both the vectorizer and the logistic regression model. Then, you pass this pipeline to the cross_val_score() while the data should the initial train dataset (not the train_cv one).

you need to set the random seed here cv=StratifiedKFold(5,shuffle = True) as well or give the same kfold for cross_val_score.

I have created a toy example for your workflow.

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.metrics import f1_score
import numpy as np

categories = ['alt.atheism', 'talk.religion.misc']
newsgroups_train = fetch_20newsgroups(subset='train',
                                      categories=categories)

from sklearn.linear_model import LogisticRegression

train, labels = np.array(newsgroups_train.data), newsgroups_train.target


kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=2020)
for train_idx, val_idx in kfold.split(train, labels):
    x_train, y_train = train[train_idx], labels[train_idx]
    x_val, y_val = train[val_idx], labels[val_idx]

    count_vectorizer = CountVectorizer()
    count_vectorizer.fit(x_train)
    X_train_cv = count_vectorizer.transform(x_train)
    X_val_cv = count_vectorizer.transform(x_val)

    cv_classifier = LogisticRegression(solver='lbfgs', C=25, max_iter=500)
    cv_classifier.fit(X_train_cv, y_train)
    y_pred = cv_classifier.predict(X_val_cv)
    f1 = f1_score(y_val, y_pred, average='macro')
    print(f1)

cv_classifier = LogisticRegression(solver='lbfgs',C=25,max_iter=500,class_weight='balance') 
classifier_pipeline = make_pipeline(CountVectorizer(), cv_classifier) 
print(cross_val_score(classifier_pipeline,train, labels, cv=kfold,scoring='f1_macro'))

output:

0.9059466848940533
0.9527147766323024
0.9174937965260546
0.9336297237218165
0.9526315789473685
[0.90594668 0.95271478 0.9174938  0.93362972 0.95263158]