We recently integrating some of our old product and doing some load testing, one of the product are using legacy tech .Net remoting (using TCP). And through Process Explorer I discovered each proxy call would create 1 TCP/IP connection.
Both remoting client and server are hosted as Window Service, and we have multiple server and each server at most will only received 4 request at the same time, so I'm expecting the limitation are coming from remoting client, and that's only 1 remoting client are doing all the task, as the client server are designed to be the core middleman.
Here's what I have research and tested so far
- Client machine seems good, CPU, memory, Network and IO still have resources to spare
- Increased min thread pool
- Increased max connection in app.config
The behavior are somewhat similar like thread pool, when active thread are more than the min thread in the setting, all the subsequence task which require thread will be put into the queue, and new thread will be created if thread pool are unable to release free thread on time, but each created thread will introduce approximate 500ms overhead.
The latency started to occur when the it reached around 250 requests, so my question is, does TCP introduce some overhead when too many connection are establishing at the same time just like thread pool?
Being researching this for many days but still no clue yet, any advice or hint will be very much appreciated.