I am new to Spark so please bare with the beginners way of explaining a problem.
I have a trained Spark MLlib model that I am trying to load into my pyspark code.
model = RandomForestClassificationModel().load('file:///home/nuser/trained_model')
But when the code runs on the cluster I get the error:
pyspark.sql.utils.AnalysisException: Multiple sources found for parquet (org.apache.spark.sql.execution.datasources.v2.parquet.ParquetDataSourceV2, org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat), please specify the fully qualified class name.;
I know that there is a way to specify the format for cases where there are multiple csv formats in the code by specifying the format (ex:org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2) like shown below:
df = spark.read.format("org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2") .option("header", "false").schema(schema_format).load(file loc).
I want to know if there is a similar way to specify the fully qualified parquet class name while loading the model to fix this issue, if not how do I remove one of the sources to get rid of the error.
- pyspark version : 3.0.1