Comparison of loading a csv file in solr-8.9.0 on different machines

user_752590 0
Viewed 113

I have two machines, On both machines I have installed solr-8.9.0. i have updated the solr core, with same csv file having size as 546MB.

Machine 1 configuration is as follows-

open file limit is 1024.
Max Processes Limit is 31168.

Memory allocated to solr-8.9.0 is 512MB. This can be checked using following command 'ps -ef|grep solr'

solr    4955  6959  1 Jul19 pts/0    00:54:51 /home/solr/installer/jdk/jdk1.8.0_66/bin/java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/solr/Lucene/solr/solr-8.9.0/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/home/solr/Lucene/solr/solr-8.9.0/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/home/solr/Lucene/solr/solr-8.9.0/bin/oom_solr.sh 8983 /home/solr/Lucene/solr/solr-8.9.0/server/logs -Djetty.home=/home/solr/Lucene/solr/solr-8.9.0/server -Dsolr.solr.home=/home/solr/Lucene/solr/solr-8.9.0/server/solr -Dsolr.data.home= -Dsolr.install.dir=/home/solr/Lucene/solr/solr-8.9.0 -Dsolr.default.confdir=/home/solr/Lucene/solr/solr-8.9.0/server/solr/configsets/_default/conf -Xss256k -Dsolr.log.muteconsole -jar start.jar --module=http

To load a csv file following command has been used.

curl 'http://localhost:8983/solr/comparison/update?commit=true&separator=|' --data-binary @/home/solr/data_546MB_7fields.csv -H 'Content-type:application/csv'

processor model name : Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz

Number of processor - 8

RAM - 8GB

On loading of file having 546M, solr-8.9.0 is taking 5 minutes 7 seconds.

Machine-2 configuration is as follows-

open file limit is 4096.
Max Processes Limit is 9216.

Memory allocated to solr-8.9.0 is 32GB. This can be checked using following command 'ps -ef|grep solr'

solr    28821     1 62 11:44 pts/1    00:16:28 /home/solr/installer/jdk1.8.0_66/bin/java -server -Xms10240m -Xmx32768m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/solr/installer/solr/solr-8.9.0/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/home/solr/installer/solr/solr-8.9.0/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/home/solr/installer/solr/solr-8.9.0/bin/oom_solr.sh 8983 /home/solr/installer/solr/solr-8.9.0/server/logs -Djetty.home=/home/solr/installer/solr/solr-8.9.0/server -Dsolr.solr.home=/home/solr/installer/solr/solr-8.9.0/server/solr -Dsolr.data.home= -Dsolr.install.dir=/home/solr/installer/solr/solr-8.9.0 -Dsolr.default.confdir=/home/solr/installer/solr/solr-8.9.0/server/solr/configsets/_default/conf -Xss256k -Dsolr.log.muteconsole -jar start.jar --module=http

processor model name : Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz

number of processor - 48

RAM - 128GB

On loading of file having size as 546M, solr-8.9.0 is taking 9 minutes 28 seconds.

Why solr running on machine-2 is taking 1.65 times of machine-1 for loading a csv file, If fact i thought machine-2 shall take less time as open file limit and memory allocated to solr is higher than than machine-1.

It is really awful.Im pretty new to this, so i have tried to post everything you need to understand my problem. Your help is much appreciated.

solr java
1 Answers

Is there an important reason to stay on the old Java 8 (and even than this old version update 66, based on the directory name?) you are missing out on a lot of improvements in later versions.

Without the results from profilers like: JDK Flight Recorder or Linux perf (with e.g. flamegraphs as visualisation) I am left to guess which of the parameters you provide may be causing a slowdown.

Try and remove the option -XX:+AlwaysPreTouch with this large heap size: -Xmx32768m because in Java 8 this was single-threaded for the G1 collector.

Also consider decreasing the max heap size if you don't need all that memory to allow for optimizations like compressed pointers (oops), the less memory you use the faster your code runs (unless the the heap is too full and you are frequently running Full-GC, or other issues such as https://plumbr.io/handbook/gc-tuning-in-practice/premature-promotion )

If you need to stay on Java 8 you should at least make sure you use the latest version (not update 66 from 5 years ago) and maybe not on G1GC that was still somewhat buggy in older releases, and instead use the default, which should be -XX:+UseParallelGC if you don't care about pause times. (If you do have interactive users that care about pause-times you may want to compare other options like CMS or the experimental Shenandoah).

To find out how much heap memory your application requires you may want to analyse your solr_gc.log with tools like: https://gceasy.io/

Sources:

user_62506490
Related