CV and under sampling on a test fold

Generally, the best way to create a cross-validation set is to simulate your test data. In your case, if we are going to divide your data into 3 sets (train, crossv., test), the best way to do it creating sets with the same proportion of true label/false label. That's what I did in the following function.

import numpy as np
import math
X=DF[["Have_0","Have_1","Have_2","Have_letters"]]
y=DF["Label"]
 


 
def create_cv(X,y):
    if type(X)!=np.ndarray:
        X=X.values
        y=y.values
 
    test_size=1/5
    proportion_of_true=y[y==1].shape[0]/y.shape[0]
    num_test_samples=math.ceil(y.shape[0]*test_size)
    num_test_true_labels=math.floor(num_test_samples*proportion_of_true)
    num_test_false_labels=math.floor(num_test_samples-num_test_true_labels)
    
    y_test=np.concatenate([y[y==0][:num_test_false_labels],y[y==1][:num_test_true_labels]])
    y_train=np.concatenate([y[y==0][num_test_false_labels:],y[y==1][num_test_true_labels:]])


    
    X_test=np.concatenate([X[y==0][:num_test_false_labels] ,X[y==1][:num_test_true_labels]],axis=0)
    X_train=np.concatenate([X[y==0][num_test_false_labels:],X[y==1][num_test_true_labels:]],axis=0)
    return X_train,X_test,y_train,y_test

    
    
X_train,X_test,y_train,y_test=create_cv(X,y)
X_train,X_crossv,y_train,y_crossv=create_cv(X_train,y_train)
    

By doing so we have sets with the following shapes (which all have the same proportion of true label/false label):

I am assuming that your test data is not representative as it is too small for this purpose (there isn't much left after splitting it a few times, as Cross-Validation is further splitting the dataset).

For under and over sampling there is a great library called imbalanced-learn. It also comes with good documentation, such as on under sampling.

Given your sample data:

from io import StringIO
import pandas as pd
from sklearn.model_selection import train_test_split
from imblearn.under_sampling import RandomUnderSampler

df = pd.read_csv(StringIO(
    """
    Have_0 Have_1 Have_2 Have_letters Label
    1        0      1         1         1
    0        0      0         1        -1 
    1        1      1         1        -1
    0        1      0         0         1
    1        1      0         0         1
    1        0      0         1        -1
    1        0      0         0         1
    """
), sep='\s+')

y = df['Label']
X = df.drop('Label',axis=1)

You could then under sample the training dataset:

under_sampler = RandomUnderSampler(random_state=0)
X_train_resampled, y_train_resampled = under_sampler.fit_resample(X_train, y_train)

You could pass that to the Cross-Validation. The downside is that it will be evaluated (as part of CV) based on the balanced dataset. That may be okay for model selection purpose though. Instead you could apply the sampling to the train split of each CV fold (as you would have to do for oversampling).

If your dataset is small, then you could benefit from using over sampling instead. In that case you need to be mindful that over sampled data is not split afterwards. That is because it would create data leakage (leading to wrong scores; see imbalance-learn's common pitfalls). That is easy to avoid with train_test_split, as you can just call the sampler after calling train_test_split. But using cross validation would lead to more splits (hidden in cross_val_score). Here the over sampling would need to happen after each CV split. You could do that using sklearn's KFold or StratifiedKFold class for example.

Something like this:

def get_train_sampled_cv_splits(train_test_indices_splits, sampler, y):
    for train_indices, test_indices in train_test_indices_splits:
        y_train_split = y.iloc[train_indices]
        train_indices_resampled, _ = sampler.fit_resample(train_indices.reshape(-1, 1), y_train_split)
        yield train_indices_resampled.reshape(-1), test_indices

over_sampler = RandomOverSampler(random_state=0)
kf = KFold(n_splits=2, shuffle=True, random_state=42)
resampled_train_test_indices_splits = get_train_sampled_cv_splits(
    kf.split(X_train, y_train),
    over_sampler,
    y
)
cross_val_score(tree, X_train, y_train, cv=resampled_train_test_indices_splits, scoring="f1")

You will also already be mindful of the metric for imbalanced datasets (accuracy is not usually a good one). Someone shared a chart Evaluation Metrics for Imbalanced Classification on Kaggle that might be useful.