I was using solr 5.5 earlier on normal mode (non-cloud). On it, I had ~3.1 million documents. The documents has high amount of textual content. It was running fine 40GB memory. Now I need to use solr in cloud mode. And since, 5.5 is old, I decided to switch to solr 9.0.0 and running it cloud mode. For data migration, I exported JSON files in few thousand chunks from solr 5.5. I imported these chunks into solr 9.0 using curl. With commit after a certain number of chunks. For indexing, the 9.0 is being ran on 100GB RAM configuration. Until now, after ~2.5 million documents have been indexed. Similar to 5.5, I optimised the collection and try running it with 40GB RAM. However, the core is not getting initialised. I tried increasing the RAM and it is working for not less than 90GB RAM. I am using CMS garbage collector. I tried tuning some options in solr configuration and try to comb the logs but to no avail. Not sure why 9.0 needs too much RAM for core to initialise. One reason I can think of is number of segments. Since, 5.5 has only 1 segment for all 3.1M documents. 9.0 has more than 100 segments. I tried merging the segments but solr is getting crashed on each run. Indexing more documents also results in crash. Any help is highly appreciated. Thanks!
These are the GC options I have added in solr.in.sh file
GC_LOG_OPTS='-Xlog:gc*' # (Java 9+)
GC_TUNE=" \
-XX:+ExplicitGCInvokesConcurrent \
-XX:SurvivorRatio=4 \
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:+CMSScavengeBeforeRemark \
-XX:PretenureSizeThreshold=64m \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=50 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled"