I am working on a binary classification problem with an imbalanced dataset where 75% of the data belongs to the negative class(0.0) and the rest (25%) belongs to the positive class(1.0).
I am using a PySpark Dataframe where each row has a label (0.0 or 1.0) associated with it for indicating the class. Due to the imbalance of the classes, I would like to use appropriate class weights.
From the documentation and the example listed here, there's a parameter called weightCol in the line
blor = LogisticRegression(weightCol="weight")
The description of weightCol is mentioned here.
So can I go ahead and create a new column called weight, assign a value of 0.75 whenever the label is 1.0 and 0.25 when the label is 0.0 for every row and initialise the model as mentioned above?
I just want to check if this is the right way to assign weights to an imbalanced dataset in Spark MLlib as the documentation doesn't make it really clear