I have a solr cloud cluster which contains:
- 3 solr instances (32GB of ram per instance, 4 cores per instance - r5.xlarge on aws)
- 1 zookeeper instnace (2GB of ram, 1 core - t2.small on aws)
- data is stored across 3 different shards with replication factor 2
- data is populated directly from the spark (aws emr) after processing from the cluster of 35 servers. (hard commit during update is happening every 30 seconds)
I have around 37 millions of records which I was indexing/(reindexing from time to time) and serving with this solr cluster - in total around 20GB of data indexed.
Everything was working fine and I never had issues with indexing. However, right now we have a data increase to 200 milions of records which is substantial increase.
When I try to index this amount of data it stuck between 50 - 100 millions of records (its randomly, never fail on exactly the same place) with the following error:
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: Request processing has stalled for 20013ms with 100 remaining elements in the queue.
at org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1118)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:195)
...
I did a research on this, and as per my understanding what is happening is that during the distributed update, one of the servers takes more time to respond back causing the process to fail.
If that is the case, what is the best way to conclude why the process stuck in general (I mean, what exactly during indexing takes the biggest amount of time)? What kind set of logs will be helpful to include to conclude this? As per your experience, what is the common reason for this behavior (if there is any).
Interesting enough, cpu usage during this process is around 70% - even when it fails.