we have a set of marketing emails that we currently regularly send out to our members. Now, we would like to see using these email campaigns, which members will likely purchase miles and which ones will not. So we extract a lot of features from these current emails like the time they were sent, day of the week they were sent, number of links in the emails, number of pictures in the emails, word counts of the emails, membership level, etc. Based on this feature space, we build a binary classification model to see which members will purchase and which ones will not. After we build the model, we observe the predictions and see some features have direct relation with members who purchased and some others like having zero images in the body emails won't help at all, so we say okay let's add more images to the email body. we add more images to the email body and send emails with more images to some members and send the same old emails with no image number changes to some others to check if it really helped or not.At the end of a period like a quarter, we see the profit increased by let's say 10%. How do I make sure this is the direct result of the changes out of the ML model and nothing else? Do I use hypothesis testing? How?
0 Answers
Related