I use haproxy as tcp balancer for my servers. There are a few hundred non-zero Send-Q connections between haproxy and clients. And there are many 'cD' flag in haproxy log. Now many server responses reach clients very slowly(more than 10 seconds). Is it caused by clients not receiving data? Or haproxy server does not work properly? Or haproxy server reaches bandwidth limit? What can I do to find the reason?
#455 non-zero Send-Q connection
ubuntu@ip-172-31-19-218:~$ netstat -atn|awk '{if($3>0) print $0}'|wc -l
455
#Top five Send-Q connections
ubuntu@ip-172-31-19-218:~$ netstat -atn|awk '{if($3>0) print $0}'|sort -k3nr|head -n 5
tcp 0 27292 172.31.19.218:12135 :47685 ESTABLISHED
tcp 0 22080 172.31.19.218:12135 :11817 ESTABLISHED
tcp 0 21886 172.31.19.218:12135 :12755 ESTABLISHED
tcp 0 21584 172.31.19.218:12135 :8753 ESTABLISHED
#many 'cD' flags in haproxy log
ubuntu@ip-172-31-19-218:/var/log$ awk '{print $12}' haproxy.log | sort | uniq -c
3
7525 --
**4687 cD**
526 CD
1 /run/haproxy.pid
3 SD
#some 'cD' flag logs
[27/Sep/2017:10:04:11.791] game nodes/s23 1/1/424425 34577 cD 4130/4130/4130/154/0 0/0
[27/Sep/2017:10:09:59.272] game nodes/s34 1/0/77777 3387 cD 4129/4129/4129/165/0 0/0
[27/Sep/2017:09:55:18.557] game nodes/s13 1/0/958654 84303 cD 4128/4128/4128/173/0 0/0
[27/Sep/2017:10:09:34.121] game nodes/s15 1/0/103309 3573 cD 4127/4127/4127/168/0 0/0
#haproxy config
ubuntu@ip-172-31-19-218:/var/log$ cat /etc/haproxy/haproxy.cfg
global
daemon
maxconn 200000
log 127.0.0.1 local0
defaults
maxconn 200000
timeout connect 5000
timeout client 60000
timeout server 60000
listen game
bind *:12135
mode tcp
option tcplog
log global
balance roundrobin
default_backend nodes
backend nodes
server s11 172.31.20.23:12137
....