I am trying to deploy a mmlspark.lightgbm model on my pyspark code. I tried pip install mmlspark on master's ssh and I got that it is satisfied, but when I run my project I got again:
mmlspark not found.
My code consists a spark session like this:
spark = SparkSession.builder.appName("MyApp").master("yarn").config("spark.sql.shuffle.partitions",20).config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1").config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven").getOrCreate()
And I import mmlspark:
from mmlspark.lightgbm.LightGBMClassifier import LightGBMClassifier
This runs locally without problems