tcp_tw_reuse vs tcp_tw_recycle : Which to use (or both)?

user_504809 0
Viewed 113719

I have a website and application which use a significant number of connections. It normally has about 3,000 connections statically open, and can receive anywhere from 5,000 to 50,000 connection attempts in a few seconds time frame.

I have had the problem of running out of local ports to open new connections due to TIME_WAIT status sockets. Even with tcp_fin_timeout set to a low value (1-5), this seemed to just be causing too much overhead/slowdown, and it would still occasionally be unable to open a new socket.

I've looked at tcp_tw_reuse and tcp_tw_recycle, but I am not sure which of these would be the preferred choice, or if using both of them is an option.

tcp networking linux
3 Answers
user_15649460

pevik mentioned an interesting blog post going the extra mile in describing all available options at the time.

Modifying kernel options must be seen as a last-resort option, and shall generally be avoided unless you know what you are doing... if that were the case you would not be asking for help over here. Hence, I would advise against doing that.

The most suitable piece of advice I can provide is pointing out the part describing what a network connection is: quadruplets (client address, client port, server address, server port).

If you can make the available ports pool bigger, you will be able to accept more concurrent connections:

  • Client address & client ports you cannot multiply (out of your control)
  • Server ports: you can only change by tweaking a kernel parameter: less critical than changing TCP buckets or reuse, if you know how much ports you need to leave available for other processes on your system
  • Server addresses: adding addresses to your host and balancing traffic on them:
    • behind L4 systems already sized for your load or directly
    • resolving your domain name to multiple IP addresses (and hoping the load will be shared across addresses through DNS for instance)
user_5731520
Related