I would like to find out if it's possible to load a saved Flux model or its weights (a .bson file) into an MLJ machine. I would like to evaluate the performance of my model, using the Performance measures API in MLJ (e.g confusion matrix, Fscore etc.). Is this possible?
I am still new to the Julia ML ecosystem, so any help or advice will be much appreciated.
Unfortunately there isn't any official way to do this. I suppose the reason you want to do this is to evaluate performance of your model without having to re-train your model. If this is the case you don't necessarily have to wrap your flux model in an MLJ machine to evaluate it. But the downside is that you won't be able to use MLJ built-in cross validation strategies (This shouldn't be a problem as using cross-validation amounts to refitting your flux model on a subset of your data, which isn't what your want anyway). Here is an example of how this this could be done.
julia> using Flux, MLJBase
julia> model = Chain(Dense(10,5,relu),Dense(5,2),softmax)
Chain(Dense(10, 5, NNlib.relu), Dense(5, 2), NNlib.softmax)
julia> using BSON: @load
# The line below assume you have previously saved "mymodel.bson"
julia> @load "mymodel.bson" weights
julia> Flux.loadparams!(model, weights);
julia> predict = model;
For the following, x_test is test feature_data, while y_test is the test target data.
julia> ConfusionMatrix()(predict(x_test), y_test);
julia> FScore()(predict(x_test), y_test);
It is also worth nothing that MLJ also supports loading pre-trained models in the form of machines, provided the machine was saved with MLJ. See MLJSerialization. To build flux models using MLJ interface checkout MLJFlux