I'm not sure if this is intended but the export.pkl from the learn.export() is about 471 MB which is somewhat prohibitive in the deployment in certain applications.
The model itself from SaveModelCallback is only 131 KB and I'm only looking to use the Learner in order to apply the same transforms/processing (Normalization, FillMissing, Categorify).
Is there a reason this is so large? I've also confirmed
learn.xb =
(None, )
learn.yb =
(None, )
Original Post: https://forums.fast.ai/t/tabularlearner-export-pkl-from-learn-export-is-very-large/81251/2
Must pip install wwf see https://walkwithfastai.com/tab.export
from wwf.tab.export import *
We manually save the Model in the Learner
torch.save(learn.model, f'{model_dir}/2_{REF}_LEARNER_MODEL.pt')
We export the Tabular Object as well
to.export(f'{model_dir}/3_{REF}_TABULAR_OBJECT.pkl')
We load the Tabular Object
to_new = load_pandas(f'{model_dir}/3_{REF}_TABULAR_OBJECT.pkl')
to_new = to_new.train.new(df[:20])
to_new.process()
model_2 = torch.load(f'{model_dir}/2_{REF}_LEARNER_MODEL.pt')
learn_new = TabularLearner(dls_new, model)
row, clas, probs = learn_new.predict(df.iloc[0])
row.show()
probs
The savings are substantial:
vs.