Solr File Descriptor Count

user_8727676 0
Viewed 1666

I have an Apache Solr 4.2.1 instance that has 4 cores of total size (625MB + 30MB + 20GB + 300MB) 21 GB.

It runs on a 4 Core CPU, 16GB RAM, 120GB HD, CentOS dedicated machine.

1st core is fully imported once a day.

2nd core is fully imported every two hours.

3rd core is delta imported every two hours.

4rth core is fully imported every two hours.

The server also has a decent amount of queries (search for and create, update and delete documents).

Every core has maxDocs: 100 and maxTime: 15000 for autoCommint and maxTime: 1000 for autoSoftCommit.

The System usage is:

  1. Around 97% of 14.96 GB Physical Memory

  2. 0MB Swap Space

  3. Around 94% of 4096 File Descriptor Count

  4. From 60% to 90% of 1.21GB of JVM-Memory.

When I reboot the machine the File Descriptor Count fall to near 0 and then steadily over the course of on week or so it reaches the aforementioned value.

So, to conclude, my questions are:

  1. Is 94% of 4096 File Descriptor Count normal?

  2. How can I increase the maximum File Descriptor Count?

  3. How can I calculate the theoretical optimal value for the maximum and used File Descriptor Count.

  4. Will the File Descriptor Count reaches 100? If yes, the server will crash? Or it will keep it bellow 100% by itself and functions as it should?

Thanks a lot beforehand!

descriptor solr count file
2 Answers
  1. Sure.
  2. ulimit -n <number>. See Increasing ulimit on CentOS.
  3. There really isn't one - as many as needed depending on a lot of factors, such as your mergefactor (if you have many files, the number of open files will be large as well - this is especially true for the indices that aren't full imports. Check the number of files in your data directories and issue an optimize if the same index has become very fragmented and have a large mergefactor), number of searchers, other software running on the same server, etc.
  4. It could. Yes (or at least it won't function properly, as it won't be able to open any new files). No. In practice you'll get a message about being unable to open a file with the message "Too many open files".
user_1376500

So, the issue with the File Descriptor Count (FDC) and to be more precise with the ever increasing FDC was that I was committing after every update!

I noticed that Solr wasn't deleting old transaction logs. Thus, after the period of one week FDC was maxing out and I was forced to reboot.

I stopped committing after every update and now my Solr stats are:

  1. Around 55% of 14.96 GB Physical Memory
  2. 0MB Swap Space
  3. Around 4% of 4096 File Descriptor Count
  4. From 60% to 80% of 1.21GB of JVM-Memory.

Also, the old transaction logs are deleted by auto commit (soft & hard) and Solr has no more performance wornings!

So, as pointed very well in this article:

Understanding Transaction Logs, Soft Commit and Commit in SolrCloud

"Be very careful committing from the client! In fact, don’t do it."

user_87276760
Related