Epoll - EPOLLONESHOT multithreaded

I'm currently building a websocket server using epoll with non-blocking sockets. The server supports SSL using OpenSSL. My setup is the following:

1. I have 1 main thread, that runs epoll_wait() on all client sockets. When something new happens (EPOLLIN | EPOLLOUT) it ships out the work to one of n worker threads. All client sockets are configured with EPOLLET | EPOLLONESHOT using EPOLL_CTL_ADD/EPOLL_CTL_MOD.
2. I have n worker threads that either: connect, disconnect, read, or write for the client socket. When one of the 4 jobs has been processed, the client socket is configured with EPOLLIN/EPOLLOUT, EPOLLET AND EPOLLONESHOT using EPOLL_CTL_MOD again.
3. With the use of EPOLLONESHOT, I guarantee that a client file descriptor is only used by one thread at a time.
4. When one client is writing to another, the message sent, is currently put into a MPSC ringbuffer of the receiver. This ringbuffer needs to be processed by the receiving file descriptor using a worker thread performing the write job. In order for the receiving file descriptor to know that it should perform a write I need to set EPOLLOUT on its epoll file descriptor.

At this point the receiving file descriptor could be doing other work such as reading or could simply be waiting in the main thread. My question is how do I (thread)safely use EPOLL_CTL_MOD on the receiving client's epoll file descriptor, such that I maintain the guarantee in point 3?

I guess that handling the case where the client file descriptor is currently doing another job, can be handled by having another MPSC ringbuffer with epoll events to be added at the end of worker processing, but how do I handle the case where no jobs are currently being processed for the file descriptor without introducing some kind of race condition?

The question is somewhat related to the workaround part of How should I use epoll to read and write from the same FD, but even using dup and having seperate waits for read and writes, I do not think is an option as OpenSSL can require SSL_ERROR_WANT_WRITE for reads and SSL_ERROR_WANT_READ for writes.