Where would my data be getting lost at within this mutex/pthread_cond_wait structure?

user_8477506 0
Viewed 55

FINAL EDIT: Solution to problem was stated by the answer I have selected. The representative example code is shown in the diff here

EDIT: Full compile-able code at the bottom of the post.

I have this rudimentary multithreaded server that simply accepts a connection and is supposed to pass the file descriptor off to a thread to allow this thread to handle it directly until the client disconnects.

For some reason, even with the following code flow inside of the server, some clients "Fall through the cracks" and get stuck in limbo. (They never get handled by the server so they just hang after accepting the connection)

The following block is my server main running loop:

    while(g_serv.b_running)
    {
        //printf("Awaiting connection.\n");
        client_fd = accept(g_serv.serv_listener_fd,
                           (struct sockaddr*)&cli_addr,
                           &clilen);
        if (0 > client_fd)
        {
            fprintf(stderr,
                    "Error accepting connection. [%s]\n",
                    strerror(errno));
            continue;
        }


        err = sem_trywait(&(g_serv.client_count_sem));
        if (0 > err)
        {
            fprintf(stderr,
                    "Max connections reached. [%s]\n",
                    strerror(errno));
            notify_client_max_connections(client_fd);
            close(client_fd);
            client_fd = 0;
            continue;
        }

        printf("A client has connected.\n");

        char byte[2] = "0";
        err = send(client_fd, byte, 1, 0);

        // Set up client FD in global position and wake up a thread to grab it
        //
        pthread_mutex_lock(&(g_serv.new_connection_fd_lock));
        g_serv.new_connection_fd = client_fd;
        if (0 != g_serv.new_connection_fd)
        {
            pthread_cond_signal(&(g_serv.new_connection));
        }
        pthread_mutex_unlock(&(g_serv.new_connection_fd_lock));
    }

This block is the thread handling function:

    void* thread_handler(void* args)
    {
        serv_t* p_serv = (serv_t*)args;
        bool    thread_client_connected;
        int     thread_client_fd;
        while(p_serv->b_running)
        {
            pthread_mutex_lock(&(p_serv->new_connection_fd_lock));
            while (0 == p_serv->new_connection_fd && p_serv->b_running)
            {
                pthread_cond_wait(&(p_serv->new_connection),
                                  &(p_serv->new_connection_fd_lock));
            }
            thread_client_fd = p_serv->new_connection_fd;
            p_serv->new_connection_fd = 0;
            pthread_mutex_unlock(&(p_serv->new_connection_fd_lock));
    
            // In the case of a pthread cond broadcast for exiting the server.
            //
            if (0 == thread_client_fd)
            {
                continue;
            }
            
            thread_client_connected = true;
            while (thread_client_connected)
            {
                thread_client_connected = handle_client(thread_client_fd);
            }
            close(thread_client_fd);
            thread_client_fd = 0;
            sem_post(&(p_serv->client_count_sem));
        }
        return NULL;
    } /* thread_handler */

Just for data reference here is my serv_t struct:

    typedef struct serv_t {
        bool            b_running;
        int             max_connections;
        int             serv_listener_fd;
        sem_t           client_count_sem;
        pthread_mutex_t new_connection_fd_lock;
        pthread_cond_t  new_connection;
        int             new_connection_fd;
        pthread_t*      p_thread_ids;
    } serv_t;

Basically, if I run netcat or a client program I have against it with multiple instances via a bash command to "background" the application, some of these instances get stuck. I have it redirecting the output to a file, but what's happening is that particular instance of the client/netcat is just getting stuck after the accept call.

More specifically, if I run my program with two threads, one instance of a program gets stuck and no subsequent copies get stuck, even running 6500 instances against the server.

If I run it with ten threads, as many as 8 or 9 instances get stuck, but the threads still function properly within the server.

EDIT:

Client code I refer to, starting from the server letting the client know that the server is ready to receive data:

    char buff[2] = { 0 };
    err = recv(client_socket_fd, buff, 1, 0);

    if ('0' != buff[0] && 1 != err)
    {
        fprintf(stderr,
                "Server handshake error. [%s]\n",
                strerror(errno));
        close(client_socket_fd);
        return EXIT_FAILURE;
    }

    if (NULL != p_infix_string)
    {
        if (MAX_BUFFER_SIZE < strlen(p_infix_string))
        {
            fprintf(stderr,
                    "Infix string is over 100 characters long.\n");
            return EXIT_FAILURE;
        }
        errno = 0;
        char* p_postfix = infix_to_postfix(p_infix_string);
        if (EINVAL == errno || NULL == p_postfix)
        {
            fprintf(stderr, "Error converting provided string.\n");
        }
        bool success = send_postfix(p_postfix, client_socket_fd);

        free(p_postfix);
        if (false == success)
        {
            fprintf(stderr,
                    "An error occured while sending the equation to the server.\n");
            close(client_socket_fd);
            return EXIT_FAILURE;
        }
    }

The client is getting stuck at the receive call here:

bool send_postfix(char* p_postfix, int client_socket_fd)
{
    if (NULL == p_postfix)
    {
        fprintf(stderr, "No postfix string provided to send to server.\n");
        return false;
    }

    printf("Sending postfix to server\n");
    int err = send(client_socket_fd,
                   p_postfix,
                   strnlen(p_postfix, MAX_BUFFER_SIZE),
                   0);
    if(strnlen(p_postfix, MAX_BUFFER_SIZE) > err)
    {
        fprintf(stderr,
                "Unable to send message to server. [%s]\n",
                strerror(errno));
        return false;
    }

    char response[MAX_BUFFER_SIZE] = { 0 };
    printf("Waiting for receive\n");
    err = recv(client_socket_fd, &response, MAX_BUFFER_SIZE, 0);
    if (0 == err)
    {
        fprintf(stderr,
                "Connection to server lost. [%s]\n",
                strerror(errno));
        return false;
    }
    else if (0 > err)
    {
        fprintf(stderr,
                "Unable to receive message on socket. [%s]\n",
                strerror(errno));
        return false;
    }

    printf("Server responded with: \n%s\n", response);
    return true;
} /* send_postfix */

EDIT: https://github.com/TheStaplergun/Problem-Code I uploaded the code to this repo and removed the need for the extraneous files I use and filled them with placeholders. You can recreate this problem using the server with the command ./postfix_server -p 8888 -n 2 and the client issue in another terminal with for i in {1..4}; do ./postfix_client -i 127.0.0.1 -p 8888 -e "3 + $i" &> $i.txt & done

The output of each client will be forcefully flushed because of the setbuf at the top of client. Run it, see if any programs hang, if not run that command again. Just type PS and see if one of them is hanging, and look at the resulting text file. You will see it is stuck at the receive call.

If you sigint the server (CTRL + C), the client that was stuck will close with a Connection reset by peer response from the server, so the server still does have that file descriptor locked up somewhere.

I believe a race condition is happening somehow, because it only happens randomly.

A curious thing is it only happens ONCE PER SERVER INSTANCE.

If I kill that hung instance and proceed to do it again 10000 times it never does another hang until the server is reset.

pthreads unix c
1 Answers

For some reason, even with the following code flow inside of the server, some clients "Fall through the cracks" and get stuck in limbo. (They never get handled by the server so they just hang after accepting the connection)

There may be other issues, but the first one I see is that main loop does not ensure that a new connection is actually picked up by any handler thread before it tries to hand off the next connection. Even if there are handler threads already blocked on the CV when a new connection is accepted, it is possible for the main server thread to signal the CV, loop back around, accept another connection, reacquire the mutex, and overwrite the new-connection FD before any handler thread picks up the previous one. The chances of that increase if you have more threads than cores.

Note that that will also interfere with your semaphore-based counting of available handlers -- you decrement the semaphore for every semaphore accepted, but you increment it again only for those that are successfully handled.

There are various ways that you could make the main server thread wait for the new connection to be picked up by a handler. One group would involve the server waiting on a CV itself, and relying on a handler to signal it after picking up the connection. Another, perhaps simpler, approach would involve using a semaphore to similar effect. But I would suggest instead not waiting, but instead creating a thread-safe queue for available connections, so that the server doesn't have to wait. That would even allow for queueing more connections than presently available handlers, if that would be useful to you.

user_24022720
Related