MSG_MORE in send of unix stream socket is not effective on the peer's recv

user_3754125 0
Viewed 376

Here is some background about the issue that I have:

I have a unix socket of type stream server_fd = socket(AF_UNIX, SOCK_STREAM, 0). On server side, the socket is listen(2)ed to via listen(server_fd, 128) and bound to an epoll handler that handles EPOLLIN. When reading from said socket (using the epoll callback), I use accept(2) to create a new socket for the client which is bound to its own epoll handling EPOLLIN | EPOLLOUT | EPOLLHUP | EPOLLERR. So far pretty standard.

Here is the problem:

Because the data on server side is dispersed through multiple sources, and the aim is for client side to get the data in neat packages, I do something with a gist like this:

void **data_portions = NULL;
size_t *sz_data_portions = NULL;
size_t cnt_data_portions = 0;
/// ... Fill the variables above based on the needs
fill_data(&data_portions, &sz_data_portions, &cnt_data_portions);
for (size_t i = 0; i < cnt_data_portions; i++) {
    int flags = 0;
    if (i < cnt_data_portions - 1) {
        flags = MSG_MORE;
    }
    send(fd_peer, data_portions[i], sz_data_portions[i], flags);
}

efficiently, server sends data_portions content one by one all with the flag MSG_MORE except for the last one with no flags. suffice to say, server successfully sends all the data.

Now lets assume the specific scenario, namely cnt_data_portions = 2 and sz_data_portions = {128, 32}. meaning that there are two calls to send(2). First one with len = 128 and flags = MSG_MORE and second one with len = 32 and flags = 0.

Because I have used the MSG_MORE for the first call, on the client side I expect to be able to read 128 + 32 = 160 bytes in one shot using recv(2). However, client side can only read 128 bytes only from the socket. This goes against the spirit of MSG_MORE. I don't understand why the client cannot read all 160 bytes in one shot.

More info:

  • sockets are non-blocking (O_NONBLOCK) allowing address reuse (SO_REUSEADDR set to 1).
  • sunpath is "/tmp/test-socket".
  • bind(2) is used to bind the socket to sunpath on server side and connect(2) is used on client side
  • SO_RCVBUF and SO_SNDBUF are both set to 1 MB
  • I put debugs one both server and clients side. I even made sure that all portions of data are sent before client attempts to read anything. Still, same issue happens.

UPDATE

The reason that I was trying to send with MSG_MORE is that I wanted to gather data from different sources into one message. For that I switched to sendmsg.

unix-socket epoll sockets c
1 Answers

From the linux unix(7) man page:

The send(2) MSG_MORE flag is not supported by UNIX domain sockets.

and also of interest:

The SO_SNDBUF socket option does have an effect for UNIX domain sockets, but the SO_RCVBUF option does not.

So there's no point in using MSG_MORE in your code; it's for TCP and UDP sockets.

Also, the number of reads on a stream (Be it a TCP socket, UNIX domain socket, pipe, etc.) isn't related to the number of writes on the far end. You have to include things like message boundaries in a higher-level protocol that uses the stream. If doing that is an issue, you might look into a SOCK_SEQPACKET unix socket instead, combined with writev(2) to send scattered data (I think that'll cause all the data to be in a single packet).

user_99521960
Related