SubsetRandomSampler does not always include all target labels when testing

For my ML-project with pytorch I am dividing my initial data set into a training and testing set, using a custom function which ensures that all labels in my original data set exist in both training and testing set:

(working_indices, working_labels, testing_indices, testing_labels,) = split_dataset_equally_random(
    target_labels=train_labels,
    percentage_to_split=100 - TEST_PERCENTAGE,
    random_seed=-1,
)


unique_testing_indices = set([label.detach().numpy().tolist() for label in testing_labels])
count_of_unique_testing_indices = {}
for entry in unique_testing_indices:
    count_of_unique_testing_indices[entry] = testing_labels.detach().numpy().tolist().count(entry)
testing_labels_from_indices = [train_labels[index] for index in testing_indices]
#count_of_unique_testing_indices = {i: count for (i, testing_labels.detach().numpy().tolist().count(i)) in unique_testing_indices}
print(f"Unique testing indices: {unique_testing_indices}")
print(f"Unique testing indices from labels: {set([index.detach().numpy().tolist() for index in testing_labels_from_indices])}")
print(f"Number of unique testing indices: {count_of_unique_testing_indices}")

For my current application that gives me the output

Unique testing indices: {0, 1, 2, 3}
Unique testing indices from labels: {0, 1, 2, 3}
Number of unique testing indices: {0: 2160, 1: 4104, 2: 3024, 3: 1080}

Now, for testing I use the following code:

test_loader = DataLoader(
            TensorDataset(feat, torch_labels),
            batch_size=self.Model.module_options["batch_size"],
            sampler=SubsetRandomSampler(testing_indices),
        )

accuracy, predictions, prediction_distributions, actual_labels = self.Model.evaluate_model(
            test_data_loader=test_loader
        )

with evaluate_model() being defined as

def evaluate_model(self, test_data_loader=None):
    """_summary_

    Args:
        test_data_loader (_type_, optional): _description_. Defaults to None.

    Returns:
        _type_: _description_
    """
    self.model.eval()
    if test_data_loader is not None:
        predictions, actuals = list(), list()
        for (inputs, targets) in test_data_loader:
            #print(f"{inputs}, {targets}")
            print(f"{set(targets.detach().numpy().tolist())}")
            inputs, targets = inputs.to(self.device), targets.to(self.device)
            # yhat, yhat_x = self.model(inputs).to("cpu")
            yhat, yhat_x = self.model(inputs)
            yhat = yhat.to("cpu")
            yhat_x = yhat_x.to("cpu")
            # print(yhat.detach().numpy(), yhat_x)
            yhat = yhat.detach().numpy()
            actual = targets.to("cpu").numpy()
            actual = actual.reshape((len(actual), 1))
            # yhat = yhat.round()
            predictions.append(yhat)
            actuals.append(actual)
        prediction_distributions, actuals = np.vstack(predictions), np.vstack(actuals)
        predictions = np.argmax(prediction_distributions, axis=1)
        acc = accuracy_score(actuals, predictions)
        return acc, predictions, prediction_distributions, actuals
    else:
        print("Test_data_loader is none")
    return -1, -1, -1, -1

Unfortunately, sometimes I run into the issue that my sampler only picks features corresponding to two or three of the four labels during testing, i.e. one or two labels will be completely omitted during the entire run, which then is also reflected in the predictions (i.e. when plotting the used labels during testing I might only get {0, 1, 2} instead of {0, 1, 2, 3} during the entire testing run).

Why is that happening, and how can I avoid it in future sessions? The problem goes away by simply re-running the evaluation function.