Im getting issues while using spark3.0 for reading elastic. My elasticsearch version 7.6.0 I used elastic jar of the same version. Please suggest a solution.
3 Answers
Spark 3.0.0 relies on Scala 2.12, which is not yet supported by Elasticsearch-hadoop. This and a few further issues prevent us from using Spark 3.0.0 together with Elasticsearch. If you want to compile it yourself, there is a pull-request on elasticsearch-hadoop (https://github.com/elastic/elasticsearch-hadoop/pull/1308) which should at least allow using scala 2.12. Not sure if it will fix the other issues as well.
It is not official for now, but you can compile the dependency on https://github.com/elastic/elasticsearch, the steps are
- git clone https://github.com/elastic/elasticsearch.git
- cd elasticsearch-hadoop/
- vim ~/.bashrc
- export JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
- source ~/.bashrc
- ./gradlew elasticsearch-spark-30:distribution --console=plain
and finally you can find .jar package in folder: "elasticsearch-hadoop\spark\sql-30\build\distributions", elasticsearch-spark-30_2.12-8.0.0-SNAPSHOT.jar is the es packages
It's officially released for spark 3.0
Enhancements: https://www.elastic.co/guide/en/elasticsearch/hadoop/7.12/eshadoop-7.12.0.html
Maven Repository: https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-30_2.12/7.12.0
Related