I have a training dataframe with input and output variables and a validation dataframe with just input variables. I need to train a model with the training dataframe and generate results in the validation dataframe, but the training dataframe is very large and may contain data that is unrelated to my validation dataframe. They are not necessarily outliers, but data that was acquired that may not be representative for my validation dataframe at that time, but that may be related to another validation dataframe that I may acquire in the future. I would like to filter the training dataframe before training a model on it.
I thought of doing the following: fit a k-means algorithm into the validation dataframe and filter my training dataframe by the proximity the instances have to the k-means-adjusted cluster centroids in the validation dataframe. I would choose a maximum distance and throw away instances with greater distance, in hopes that this would remove points that would have nothing to do with my validation dataframe.
I ran some tests and got this result:
Before filtering (training - blue; validation - orange):
After filtering (training - blue; validation - orange):
However, I can't remove all the unrelated points and I'm throwing away points that are close to the centroids. The way I calculated the distance between the centroids and the instances was based on this post: Distance between nodes and the centroid in a kmeans cluster?
Where, after calculating the distances (sqdist), I remove the points with distances greater than a given value.
My question is: would this be the best way to do this? Is there another way to select these instances? As a final result, I would expect a blue dot cloud that would be the same shape as the orange dot cloud, but a little wider.
Note: I cannot provide my data, examples will need to be made with synthetic data.