Collate probabilities, predictions, coefficients from multiple samples of data using sklearn

Question

I would like to combine model probabilities for class 1 predictions for ALL rows from multiple (random) splits/samples of data into a single dataframe in python.

I realize that not all rows will be selected in each split, but if data sampling is replicated enough times, each row will have been selected a few times at least and model probabilities generated.

My current approach basically creates multiple test-train splits (5 in example below), and collates probabilities from each training instance into a single dataframe as shown in below code with a mock dataset:

import pandas as pd
import numpy as np

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split


####Instantiate logistic regression objects
log = LogisticRegression(class_weight='balanced', random_state = 1)

#### import some data 
iris = datasets.load_iris()

X = pd.DataFrame(iris.data[:100, :], columns = ["sepal_length", "sepal_width", "petal_length", "petal_width"])
y = iris.target[:100,]

# start by creating the first column of probs table
probs_table = pd.DataFrame(X.index, columns=["members"])

# iterate over random states while keeping track of `i`
for i, state in enumerate([11, 444, 21, 109, 1900]):
    train_x, test_x, train_y, test_y = train_test_split(
        X, y, stratify=y, test_size=0.2, random_state=state)
    pd.DataFrame(log.predict_proba(test_x)[:, 1])  #fit final model 

    probs_table[f"iter_{i+1}"] = pd.DataFrame(log.predict_proba(test_x)[:, 1])
    
probs_table

Unfortunately, I am not getting probabilities for all rows in the dataframe. Can somebody please guide me to the solution to this problem? And it would be ideal to include additional model outputs such as predictions, coefficientts for each iteration/data row.

Any other way to sample the data (i.e., other than test-train splitting) is fine as well as long as probabilities can be assembled for all dataframe rows.

user_3791640 · Answer

There are a couple problems with the code as is:

.fit() is never called here. I'm assuming you'd like it fit right after the train/test split line and before the predict_proba() call?
When you place the values into the dataframe, you're creating a new column and I assume you want one column for all iterations while keeping track of which iteration it came from in each column?

Here is code that I believe accomplishes what you'd like. It 1) loops over each random state integer, 2) creates a new train/test split, 3) fits a new model each time, and 4) predicts on each test set row.

I also have it keep track of the original index so you can see how many times each original row ends up in the prediction data frame:

EDIT: Include the coefficients as a column

import pandas as pd

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

####Instantiate logistic regression objects
log = LogisticRegression(class_weight='balanced', random_state = 1)

#### import some data 
iris = datasets.load_iris()

X = pd.DataFrame(iris.data[:100, :], columns = ["sepal_length", "sepal_width", "petal_length", "petal_width"])
y = iris.target[:100,]

dfs = []
# iterate over random states while keeping track of `i`
for i, state in enumerate([11, 444, 21, 109, 1900]):
    train_x, test_x, train_y, test_y = train_test_split(
        X, y, stratify=y, test_size=0.2, random_state=state)
    log.fit(train_x, train_y)
    preds = log.predict_proba(test_x)[:, 1]
    orig_indices = test_x.index
    df = pd.DataFrame(data={
        "orig_index": orig_indices, 
        "prediction": preds, 
        "iteration": f"iter_{i+1}",
        "coefficients": [log.coef_[0]] * len(preds)})    
    dfs.append(df)

probs_table = pd.concat(dfs)
probs_table

Collate probabilities, predictions, coefficients from multiple samples of data using sklearn

1 Answers