how to tune linux network buffer size

user_6673012 0
Viewed 4558

I'm reading "Kafka The Definitive Guide", in page 35 (networking section) it says :

... The first adjustment is to change the default and maximum amount of memory allocated for the send and receive buffers for each socket. This will significantly increase performance for large transfers. The relevant parameters for the send and receive buffer default size per socket are net.core.wmem_default and net.core.rmem_default...... In addition to the socket settings, the send and receive buffer sizes for TCP sockets must be set separately using the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem parameters.

why we should set both net.core.wmem and net.ipv4.tcp_wmem?

kernel sockets linux
1 Answers

Short answer - r/wmem_default are used for setting static socket buffer sizes, while tcp_r/wmem are used for controlling TCP send/receive window size and buffers dynamically.

More details: By tracking the usages of r/wmem_default and tcp_r/wmem (kernel 4.14) we can see that r/wmem_default are only used in sock_init_data():

void sock_init_data(struct socket *sock, struct sock *sk)
{
        sk_init_common(sk);
        ...
        sk->sk_rcvbuf           =       sysctl_rmem_default;
        sk->sk_sndbuf           =       sysctl_wmem_default;

This initializes the socket's buffers for sending and receiving packets and might be later overridden in set_sockopt:

int sock_setsockopt(struct socket *sock, int level, int optname,
                    char __user *optval, unsigned int optlen)
{
        struct sock *sk = sock->sk;
        ...
        sk->sk_sndbuf = max_t(int, val * 2, SOCK_MIN_SNDBUF);
        ...
        sk->sk_rcvbuf = max_t(int, val * 2, SOCK_MIN_RCVBUF);

Usages of tcp_rmem are found in these functions: tcp_select_initial_window() in tcp_output.c and __tcp_grow_window(), tcp_fixup_rcvbuf(), tcp_clamp_window() and tcp_rcv_space_adjust() in tcp_input.c. In all usages this value is used for controlling the receive window and/or the socket's receive buffer dynamically, meaning it would take the current traffic and the system parameters into consideration. A similar search for tcp_wmem show that it is only used for dynamic changes in the socket's send buffer in tcp_init_sock() (tcp.c) and tcp_sndbuf_expand() (tcp_input.c).

So when you want the kernel to better tune your traffic, the most important values are tcp_r/wmem. The Socket's size is usually overridden by the user the default value doesn't really matter. For exact tuning operations, try reading the comments in tcp_input.c marked as "tuning". There's a lot of valuable information there.

Hope this helps.

user_16196950
Related