How to apply Leave-one-Group-out cross validation in sklearn?

user_7657749 0
Viewed 2194

I am building a Naive Bayes classifier (nb) using sklearn.

The dataset consists of 4 subjects with each a different amount of labeled data.

I want to apply leave-one-subject-out cross validation, but I do not find a comparable example on the internet.

My data consists of the following:

x = [[2,0],[3,1],[2,1],[3,2]], [[4,2],[5,3],[5,2],[5,3]], [[7,3],[6,2],[7,1],[6,2]], [[2,3],[2,4],[3,4],[2,3]]]
y = [[0,1,3,2],[1,2,3,2],[0,1,1,1],[0,1,2,1]]

So the data of each subject is a subarray in x with the corresponding subarray of y. The input features consists of 2 elements each (for example the mean and std of an accelerometer).

I found on the internet the example of

sklearn.model_selection.LeaveOneOut

But this does not work in my example since I want to take data of a whole subject as a test set.

Is there a good equivalent for my needs?

scikit-learn validation python-3.x python
1 Answers

The sklearn's method LeaveOneGroupOut is what you're looking for, just pass a group parameter that will define each subject to leave out from the train set. From the docs:

Each training set is thus constituted by all the samples except the ones related to a specific group.

to adapt it to your data, just concatenate the list of lists.

import itertools
from sklearn.model_selection import LeaveOneGroupOut

joined_x = list(itertools.chain.from_iterable(x))
joined_y = list(itertools.chain.from_iterable(y))

logo = LeaveOneGroupOut()
for train, test in logo.split(joined_x, joined_y, groups=joined_y):
    print("%s %s" % (train, test))
>>>
[ 1  2  3  4  5  6  7  9 10 11 13 14 15] [ 0  8 12]
[ 0  2  3  5  6  7  8 12 14] [ 1  4  9 10 11 13 15]
[ 0  1  2  4  6  8  9 10 11 12 13 15] [ 3  5  7 14]
[ 0  1  3  4  5  7  8  9 10 11 12 13 14 15] [2 6]

In the first training set group 0 is the test, the second group 1 and so on.

EDIT

As @JanDM requested, to use it with cross_val_score the groups parameter should be passed as it is pass to the split() method of the crossvalidator cv

import itertools
from sklearn.model_selection import cross_val_score
cross_val_score(estimator, joined_x, joined_y, cv=logo, groups=joined_y)
user_124833460
Related