How to gracefullly close subprocesses

user_5095986 0
Viewed 48

I am new to signals/operating systems so please bear with me.

I am given a task to gracefully skill subprocesses. Let's say transformer is a service being run by systemd. When I restart the service via sudo systemctl restart xyz@9090.service I am getting this following output trace, it looks like memory address for the subprocesses. Please can you help me understand how can I gracefully exit this.

More about the program. (Main program -> one service(TCPServer) running as a thread -> Several pools of pools (multiprocessing pools) (KINDA custom written) Output of stack trace is attached here.

Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 4  ScalableBloomFilter::which_iter(char const*)
Apr 15 22:34:31 ip-5555-5553 transform_kafka.py[2963]: 5  ScalableBloomFilter::contains(char const*)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 6  /usr/local/lib/python2.7/dist-packages/scalablebloomfilter.so(+0x3e97)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 0  Utilities::logTrace(unsigned int, int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 7  /usr/bin/python(PyEval_EvalFrameEx+0x56a)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 1  QuitHandler::quit(int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 8  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 9  /usr/bin/python(PyEval_EvalFrameEx+0x5c6c)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2  /lib/x86_64-linux-gnu/libc.so.6(+0x3f040)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 10  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 3  /usr/bin/python(+0xcdae2)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 11  /usr/bin/python(PyEval_EvalFrameEx+0x5c6c)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 4  /usr/bin/python(PyDict_GetItem+0xa1)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 12  /usr/bin/python(PyEval_EvalFrameEx+0x52f6)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 5  /usr/bin/python(PyDict_GetItemString+0x24)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 13  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 6  /usr/bin/python(+0xde26f)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2021-04-15 22:34:31        CRITICAL         Common.cpp:logTrace():170        14  /usr/bin/python(PyEval_EvalFrameEx+0x569e)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2021-04-15 22:34:31        CRITICAL         Common.cpp:logTrace():170        Obtained 20 stack frames.
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 7  /usr/bin/python(PyArg_ParseTupleAndKeywords+0x10b)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 15  /usr/bin/python(PyEval_EvalFrameEx+0x52f6)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: Obtained 20 stack frames.
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 8  /usr/local/lib/python2.7/dist-packages/simplejson/_speedups.so(+0x8acf)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 16  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2021-04-15 22:34:31        CRITICAL         Common.cpp:logTrace():170        Obtained 20 stack frames.
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 9  /usr/bin/python(PyEval_EvalFrameEx+0x54f0)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 17  /usr/bin/python(+0x10de6c)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 0  Utilities::logTrace(unsigned int, int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 10  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 18  /usr/bin/python(PyObject_Call+0x3e)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 0  Utilities::logTrace(unsigned int, int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 1  QuitHandler::quit(int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 0  Utilities::logTrace(unsigned int, int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2021-04-15 22:34:31        CRITICAL         Common.cpp:logTrace():170        11  /usr/bin/python(PyEval_EvalFrameEx+0x569e)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 19  /usr/bin/python(PyEval_EvalFrameEx+0x26b7)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2021-04-15 22:34:31        CRITICAL         Common.cpp:logTrace():170        Obtained 20 stack frames.
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 1  QuitHandler::quit(int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2  /lib/x86_64-linux-gnu/libc.so.6(+0x3f040)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 1  QuitHandler::quit(int)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 12  /usr/bin/python(PyEval_EvalCodeEx+0x4ba)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: Obtained 20 stack frames.
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2  /lib/x86_64-linux-gnu/libc.so.6(+0x3f040)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 2  /lib/x86_64-linux-gnu/libc.so.6(+0x3f040)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 13  /usr/bin/python(PyEval_EvalFrameEx+0x569e)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 3  BloomFilter::insert(char const*)
Apr 15 22:34:31 ip-5555-555 transform_kafka.py[2963]: 0  Utilities::logTrace(unsigned int,

Currently we have something like signal.signal(signal.SIGTERM, lambda _signal, _frame: x.stop())

def stop(self):
        """Stop running"""
        self.logger.info("Stopping %s", self)
        self.__running = False
        self.consumer.close()
        self.producer.flush()
        self.__pool.terminate()
systemd operating-system signals multiprocessing python
1 Answers

We need to have a signal handler for the subprocesses too, added in the pool initializer.

def init_stuff():
    signal.signal(signal.SIGTERM, signal.SIG_IGN)


p = multiprocessing.Pool(initializer=init_stuff)

Now when we kill the main thread a SIGTERM is sent to the subprocesses which

user_50959860
Related