I am currently switching from spark 2.4 to spark 3. My pipeline is in scala (switching from 2.11 to 2.12 as part of the migration).
I have several LightGBM Models in production that were trained on spark 2.4. I am currently switching the package for LightGBM using: val mmlspark = "com.microsoft.ml.spark" %% "mmlspark" % "1.0.0-rc3-24-495af3e4-SNAPSHOT" which is the official release from Azure for MMLSpark on Spark 3.
I am trying to load my models on spark 3 and it is currently failing.
I get this error:
Exception in thread "main" java.lang.IllegalArgumentException: Cannot recognize JSON metadata: {"class":"com.microsoft.ml.spark.lightgbm.LightGBMClassificationModel","timestamp":1580309847587,"sparkVersion":"2.4.3","uid":"LightGBMClassifier_510af2ce3f4f","paramMap":"{}"}.
I am hoping this would not mean retraining my models again. Has anyone experienced this issue before? As this is pretty recent, I am hoping someone has already thought about this, since the documentation is pretty small at this moment (May 2021).
Thanks in advance!