How could i get an object correctly from database in a different server in python?

Viewed 32

I am trying to work with a machine learning and a web server with django, and both my web and machine learning server use the same dockerized postgres database. Machine learning server creates machine learning models and writes them to the database by using pickle. I have a Regressor class in machine learning server like this:

class Regressor:
    def __init__(self):
        self.predictions = []
        self.yTrue = []
        self.predDiffs = []
        self.model = object
        self.name = ""
        #some methods below

and i have a class that creates an appropriate database table for orm technique

class RegressorTable(DeclarativeBase):
    __tablename__ = "ml_models"

    id = Column(Integer, primary_key=True)
    regressorName = Column(String)
    regressorModel = Column(LargeBinary)

and both in machine learning and django web server, i have 2 functions in order to convert models to binary and binaries to model

def modelToBinary(model):
    return pickle.dumps(model)


def binaryToModel(data):
    return pickle.loads(data)

Here is my method in the machine learning server to save the Regressor models for the database:

def saveModelsToDb(predictorArray):
    for predictor in predictorArray:
        predictorForTable = RegressorTable()
        predictorForTable.regressorName = predictor.name
        predictorForTable.regressorModel = modelToBinary(predictor)
        DBSession.add(predictorForTable)
        DBSession.commit()

The predictorArray in the above code contains Regressor objects. When i try to save and load models in machine learning server, i can successfully save, load and use my Regressor objects. And i have a web server and here is my models:

class RegressorTable(models.Model):
    regressorName = models.CharField(max_length=50)
    regressorModel = models.BinaryField()
    
    class Meta:
        db_table = "ml_models"

and in views.py, i have modelToBinary and binaryToModel methods i mentioned above. And i have a code like this:

def getPredictions(request):
    predictorArray = RegressorTable.objects.filter()
    finalPrediction = finalDecision(predictorArray, np.array([20]).reshape(-1, 1))
    context = {"prediction": finalPrediction}
    return render(request, "get_predictions.html", context=context)

I can successfully save, load and use the Regressor objects in machine learning server. In machine learning server, i save the Regressor objects and i want to use them in my django web server. When i try to get them with above code, it throws an exception

Can't get attribute 'Regressor' on <module '__main__'>

How can i load my Regressor objects which already saved to database by machine learning server from database correctly in my django web server?

1 Answers

The module and class names need to be the same in both apps since pickle uses the names to figure out how to revive a class instance.

Based on on <module '__main__'>, your Regressor class is defined in the main module executing, which is never what happens with Django.

Define the class in e.g. regressor.py (or better yet, something like my_ml_library/regressor.py where my_ml_library is the same package in both apps) – and also remember that pickle doesn't save the function definitions, so if you make code changes that might be incompatible with old data, you would need to re-train your model.

Related