I have performed feature reduction of 105 radiomic features on 207 subjects, so my input has a shape of (207,105). Before any processing, I scaled the feature using StandardScaler with mu=0, sigma=1 since the values of these features have a very large range. After feature reduction, I have 14 features left which are combinations of the 105 features and thus differ from the original values in the input. My output has the shape of (207,14). I want to invert the transformation to perform some statistical analysis with the new values, but since the shapes are different, this gives me an error using inverse_transform.
I read something about making a custom Scaler, but this hasn't worked for me. The following is my code:
df = pd.read_excel(r'H:\input.xlsx') # Load in dataset
scaler = StandardScaler() # Scale DataFram
X = df.iloc[:,1:] # Radiomic features
y = df.iloc[:,0] # Metastases (1) or not (0)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=40) # Define StratifiedKFold with 5 splits
for train_index, test_index in skf.split(X, y):
X_train, X_test = X.loc[train_index], X.loc[test_index] #Create train- and test set for radiomic features, stratify based on the ratio of metastases
y_train, y_test = y.loc[train_index], y.loc[test_index] #Create train- and test set for metastases, stratify based on the ratio of metastases
X_train_radiomic=pd.DataFrame(scaler.fit_transform(X_train.iloc[:,6:]))
X_test_radiomic = pd.DataFrame(scaler.fit_transform(X_test.iloc[:,6:]))
rank = X_train_radiomic.corr(method='spearman') # performs Spearman rank correlation on radiomic features, discards first column with metastases
agglo=FeatureAgglomeration(n_clusters=40).fit(rank) # clusters the features into 40 clusters and fits to Spearman rank correlation
X_trainred=pd.DataFrame(agglo.transform(X_train_radiomic)) # Transforms radiomic features based on clustered features
X_testred=pd.DataFrame(agglo.transform(X_test_radiomic)) # Transforms radiomic features based on clustered features
X_train_new_inverse = scaler.inverse_transform(X_trainred)
X_test_new_inverse = scaler.inverse_transform(X_testred)
Can someone help me out with this issue?