I’m kind of a beginner in machine learning and trying to solve a classification problem. I’m working on a very unbalanced (sequential) data set (only 2% positives out of 20k records) and I'm using LSTM/GRUs to model in python using tensorflow.
This is what I do: Load the data. Split the data set into 3 data sets : A for Training (70%) B for Validating (15%) C for Testing (15%); For each data set (A, B, C), do an over-sampling on positive class in order to increase the percentage of positives from 2% to 30%. This gives me 3 new datasets which are more balanced: A’, B’, C’
Then I train my model with the data set A’ using GRU.
My Objective: Get highest F score on my test set C (Are there any better metrics? From what I've seen, F score depends on data distribution i.e. how skewed the data is. If my actual negatives to positives ratio increases then precision decreases (due to increase in False negatives) though recall more or less remains the same; and so my overall F score goes down)
My questions are:
Can I use cross entropy as my cost function while training ? (I am not altering the cost function to be more sensitive to false positives since I have already over sampled my positives )
Which data set do I have to use for validation ? B or B’ ? and what metric should I use to plot the validation learning curve to know the point where my model is over fitting? (Currently I am using accuracy on both A’ (train) and B’ (validation) to see if there is any over fitting. But it seems accuracy on B’ and f score on B doesn't correlate much. Because ultimately I want good f score on C which means I need the model which gives good f score on B)
Thanks in advance for your time! Regards.