I'm running a Gradient Boosting model that takes around 11 hours to train. I'm using a pre-emptible instance on Google Cloud to do this (because it's way cheaper). The problem is, I keep losing my progress because the instance gets terminated before the model is able to finish training.
Is there a way to checkpoint or save the progress of a partially trained model to disk? such that the training should resume from where it left off when I rerun the code.
I know there's the pickle module which allows me to dump objects on disk. But I'm not looking to save a fully trained model, but a partially trained one.