I have a dataset where I am trying to predict a binary outcome. I have a built an SVM model but am concerned by the accuracy, sensitivity, and specificity values as they seem too high to be plausible. I am relatively new to coding and machine learning. I am assuming I have made a mistake in my approach and was wondering if anyone could identify any potential issues with my code.
I am working with a dataset that has approximately 10,000 rows and 39 columns. My first step was to split into train, validate, and test dataframes since I will be comparing multiple models:
spec = c(train = .6, test = .2, validate = .2)
g = sample(cut(
seq(nrow(subset)),
nrow(subset)*cumsum(c(0,spec)),
labels = names(spec)
))
res = split(subset, g)
The predictor variable is just almost split equally among 0 and 1 in the original dataset. I have confirmed that this same ratio is maintained in the train, validate, and test sets so there is no class imbalance.
I have then tuned and built the svm model using the e1071 package in R. I am using a linear kernal and type C-classification:
tune.out <- tune(method = 'svm', YNPGALL120~., data = res$train, ranges = list(cost = c(0.001, 0.01, 0.1, 1, 5, 10, 100)))
bestmod <- tune.out$best.model
svm_model <- svm(YNPGALL120 ~ ., data = res$train, type = 'C-classification', kernel = 'linear', C = 5, scaled = TRUE)
The bestmod results in a cost of 5 and 967 support vectors. I then use the best model to make predictions using the validation set which produces a confusion matrix with values of .9781 for accuracy, .9854 for sensitivity, and .9713 for specificity:
predicted_svm <- predict(tune.out$best.model, newdata = res$validate)
confusionMatrix(res$validate$YNPGALL120, predicted_svm)
The high performance values seem too good to be true especially since I have run a logistic regression model on the same dataset and came up with accuracy, specificity, and sensitivity metrics in the mid 50s.
Is there anything wrong I've done in my steps that could lead to such high performance metrics? Alternatively, are there any good resources anyone could point me to in regards to building and tuning SVM models in R that might help?