JavaEsSpark.saveJsonToEs(JavaRDD<String>, HashMap<String, String>)
taking too much time to process each element. I have 50000 elements and it takes hours to process all of them. I have below configuration.
ES_BATCH_SIZE_BYTES = 1mb,
ES_BATCH_SIZE_ENTRIES = 1000,
ES_BATCH_WRITE_REFRESH = false,
ES_BATCH_WRITE_RETRY_COUNT = 10,
ES_BATCH_WRITE_RETRY_WAIT = 30s
. I tried to find solution from thread - SPARK-7122 which suggest updating Spark version to 1.5 or above and I am already using 1.6
My cluster or node are not overloaded and they are healthy.