I'm running 2 docker containers(TcpServer,TcpClient). In each container, there's an init.sh script which launches the applications. In the init.sh script, I've handled SIGTERM but I'm not doing any sort of handling for that(I'm not passing it to my application).
trap 'true' SIGTERM
After startup, a tcp connection is established between TcpServer and TcpClient.
TcpClient is a multi-threaded application, with 1 thread(receiver) doing:
while(true) {
//blocking tcp receive function call
//process received data or received error code.
}
So basically, the idea is that the receiver thread would always get to know about server going down 'cleanly'.
The observation is that, most of the times, when I issue 'docker stop serverContainer', the client application receives TCP 'FIN' packet after about 10 seconds. This is as per my expectations because docker first tries to kill via SIGTERM but since that is handled it then issues SIGKILL which it does only after about 10 seconds.
My current understanding is that, whenever sigkill/unhandled-sigterm is given to a process, the kernel will terminate that process and close all file descriptors opened by that process. If this is true, then I should always see a FIN packet going from server to client as soon as the process is killed.
However, a few times, FIN packet is not observed in the traces captured on both client and server end. As a result, the client doesn't get to know about the server getting down for a longer time(until it tries to send some data on that connection or the TCP's keepalive mechanism kicks in).
I'm not sure how this happens because if I explicitly issue SIGKILL to pid 1 of my server's container(from outside), then I've always seen the FIN packet. So why sometimes, and only when using docker stop, does server not send TCP FIN?
Basically I want to ask 2 things:
In Linux, when SIGKILL is issued to a TCP server process, is it guaranteed that the kernel/tcp stack will send TCP FIN packet to client before terminating?
When I use 'docker stop' how exactly are the processes spawned by the main process(PID 1 inside container) terminated? Because from what I had read, the SIGTERM/SIGKILL is given only to PID 1? So why are its child processes not adopted by init/systemd as happens otherwise(killing the parent process created outside the container).
Operating System: Red Hat Enterprise Linux Server release 7.7 (Maipo) Docker version: Docker version 19.03.14, build 5eb3275d40