Stopping a Python Multiprocessing BaseManager 'serve_forever' server?

user_108574 0
Viewed 2760

I have the following setup in Python 3.6 for interprocess communication:

from multiprocessing.managers import BaseManager
class MyManager(BaseManager): pass
MyManager.register('get_instance', callable=lambda:my_instance)
m = MyManager(address=('', 50000), authkey=b'key')
s = m.get_server()
s.serve_forever()

To avoid blocking my message loop of a larger application, I used a thread to contain this setup. So s.serve_forever() actually runs inside a run function of a thread.

This is done according to the documentation. And the setup itself worked fine with client managers calling into the shared instance.

However, I found no way to stop this 'serve_forever' server. Since the documentation didn't mention this, I checked the source code. There's a stop_event which supposedly I can set to quit the loop. But it didn't work, as the accepter daemon/thread is still running. And I can't call shutdown on the server object because I don't have the socket object c.

So how do I shut down this server?

ps. Using BaseManager.start() isn't really an option for me because the server in my case shares an async message loop which can only be accessed by the starting process. BaseManager.start() spawns a new process which no longer has access to the message loop. get_server().serve_forever() on the other hand, runs within the calling process.

python-multiprocessing python-3.x process multiprocessing python
3 Answers

Try this in the server:

import threading

s = m.get_server()
stop_timer = threading.Timer(1, lambda:s.stop_event.set())
MyManager.register('stop', callable=lambda:stop_timer.start())
s.serve_forever()

And in the client:

MyManager.register('stop')
m.stop()

UPDATE:

I solved the timeout issue by delaying the stop_event.set() with a threading.Timer

OLD:

However, across machines, you then run into long timeouts as the stop() method fails to receive data, this answer talks about that - but I could not get it work:

properly disconnect multiprocessing remote manager

I'm new to the multiprocessing managers, I will update this answer if I find a better solution. (UPDATE: Found, updated)

user_99580410

Here is a hack after reading the multiprocessing.managers source, stripped from _finalize_manager(), basically creating a connection to the server and send a shutdown msg

from multiprocessing.managers import dispatch,listener_client

_Client = listener_client['pickle'][1]
# address and authkey same as when started the manager
conn = _Client(address=('127.0.0.1', 50000), authkey=b'key')
dispatch(conn, None, 'shutdown')
conn.close()
user_178893850

you can wrap/use the manager start/shutdown to avoid this running forever server (start() starts a server for you):

m = MyManager(address=('', 50000), authkey=b'key')
m.start()
# do something
m.shutdown()
user_13797620
Related