pytorch predictions different between train and predict

I am training a pytorch model, which is itself an attribute of larger class containing the trained model plus other objects needed for inference and diagnostics. The predictions from the final epoch of my training are not matching the predictions from inference, given the same inputs.

I define a torch model class, something like this:

class torch_model_basic(nn.Module):
    def __init__(self, some_args):
        ...
    def forward(self, some_args):
        ....
    return x

My custom class that uses the torch model class looks something like this:

class MyClass(object):
    def __init__(self, config_file_path):
        self.device = 'cuda' if torch.cude.is_available() else 'cpu'
        ...

    def make_other_attribs(self, more_args):
        ...
        return more_attribs

    def train_one_epoch(self, my_model, optimizer, some_other_args):
        ...
        return model, training_input_data, training_predictions

    def train(self, args, torch_data_set):
        model = torch_model_basic(args=args) # see how we use it?
        optimizer = ...
        for epoch in args.epochs:
            model, inputs, predictions = self.train_one_epoch(model, optimizer, some_other_args)
        self.model = model.to('cpu') # make the trained model an attribute of our larger object
        self.training_predictions = predictions # so I can look at the predictions after training
        self.more_attribs = self.make_other_attribs() # lots of other stuff happens here, unrelated to training
        return self

    def predict(self, args, torch_data_set):
        model = self.model
        model.to_device(self.device).eval()
        ... #build the x_pred inference tensor
        with torch.no_grad():
            preds = model(x_pred)
        return preds

To train & save my object, I do something like this:

my_object = MyClass()
my_dataset = ... # build the training data
my_model = my_object.train(args, my_dataset)
with open('file/path/model.pickle', 'wb') as handle:
    pickle.dump(my_model, handle, protocol=pickle.HIGHEST_PROTOCOL)

To predict, it's very similar:

with open('file/path/model.pickle', 'rb') as handle:
    my_model = pickle.load(handle)
my_dataset = ... # build the prediction data, which is the same as the training data
new_predictions = my_model.predict(args, my_dataset)

I have a bunch of print statements to watch the data as it gets read in, gets transformed, gets fed to the model, and the resulting predictions. I do this for both training & prediction.

After the model has been trained, I then feed the training data back into the prediction method. I can confirm the step immediately before pred = model(x_pred) has a tensor that duplicates the training data. But the predictions return from the prediction call do not match the predictions that I persist from the train() call.

I am training on a GPU and inferring on a CPU, if that matters. And I am using AWS SageMaker to do the training/inference on remote containers. Which, by the way, makes debugging super tough. The training call saves object to S3; the prediction routine picks up the saved object from S3 in the same place.

Any obvious reason why I shouldn't expect the predictions during training (from the final epoch) to be the same as the predictions during inference to be the same, given equivalent input data? I'm new to OOP, so pointers on why what I'm doing is a bad thing are also welcome.