Reading of netcdf files in Spark 3 through Scala

user_926918 0
Viewed 202

I have used SciSpark for reading numerous netcdf files in Spark 2.x. However, I am unable to compile it for Spark 3.x (due to various types of errors such as cannot be applied to (Array[Int]), found : Array[Long] with required: Array[Int], cannot be applied to (AnyVal), etc.). Unable to even conclude if it is compatible with Spark 3. Could anyone please suggest if:

  • SciSpark is compatible with Spark 3, and

  • Any alternative to SciSpark in Spark 3.x?

apache-spark scala
2 Answers
user_153100780

If you already have a Spark cluster you can use Apache Sedona to read netcdf files. Documentation about netcdf files is sparse, but you can ask questions in their mailing list.

user_41649240
Related