Spark ElasticSearch EsHadoopIllegalArgumentException unable to find keystore with valid URI

user_10342244 0
Viewed 887

I'm trying to connect spark to my elasticsearch with SSL.

Setup

Spark 2.4.0 from CDH 6.3.2 (Cloudera)
ElasticSearch 7.6.1 (Open Distro)
elasticsearch-hadoop-7.6.1.jar

Considering

1) I already managed to authenticate logstash with SSL and pkcs12 keystore manually created
2) Connexion Spark to ES works without security

Here spark conf provided :

spark.es.nodes=node1
spark.es.port=9200
spark.es.net.ssl=true
spark.es.net.ssl.keystore.location= ===> See below what i tried
spark.es.net.ssl.keystore.type=PKCS12
spark.es.net.ssl.cert.allow.self.signed=true
spark.es.net.http.auth.user=admin
spark.es.net.http.auth.pass=admin
spark.es.nodes.wan.only=false  //tried true

Doing

spark.read.format("org.elasticsearch.spark.sql")
    .option("es.query", "?q=*:*")
    .load("spark/docs")
    .show

====================================================

FileSystem Values tried with spark.es.net.ssl.keystore.location (after copying admin.pkcs12 on all nodes)

file:///PATH/certs/admin.pkcs12

Error :

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
  ... elided
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Get Key failed: null
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:175)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:160)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:129)
  at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
  at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
  at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
  at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:685)
  at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:664)
  at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:432)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
  at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
  at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:745)
  at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)
  ... 61 more
Caused by: java.security.UnrecoverableKeyException: Get Key failed: null
  at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:435)
  at java.security.KeyStore.getKey(KeyStore.java:1023)
  at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133)
  at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)
  at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyManagers(SSLSocketFactory.java:217)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:173)
  ... 78 more
Caused by: java.lang.NullPointerException
  at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:374)
  ... 84 more

====================================================

I copied a keystore a valid admin.pkcs12 to hdfs => /user/company/ with 777 rights, (as i'm writing, is it too permissive, like ssh ?)

//returns true
FileSystem.get(spark.sparkContext.hadoopConfiguration).exists(new Path("hdfs://namenode:8020/user/company/admin.pkcs12"))

HDFS Values tried with spark.es.net.ssl.keystore.location

hdfs:///namenode:8020/user/company/admin.pkcs12
hdfs://namenode:8020/user/company/admin.pkcs12
/user/company/admin.pkcs12

Error :

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
  ... elided
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Expected to find keystore file at [...] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:175)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:160)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:129)
  at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
  at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
  at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
  at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
  at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:685)
  at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:664)
  at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:432)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
  at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
  at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
  at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:745)
  at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)
  ... 61 more
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Expected to find keystore file at [...] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:195)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyManagers(SSLSocketFactory.java:215)
  at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:173)

I tried JKS too. What am I missing ?

elasticsearch apache-spark ssl
1 Answers
//Works
file:///PATH/certs/admin.pkcs12

I was getting this error because of the missing password.

spark.es.net.ssl.keystore.pass=PASSWORD
user_103422440
Related