My program involves a blocking loop. I'd like to use the a python REPL to modify program state while the program is running.
Context:
Often I have a blocking loop in my python code:
# main.py
import itertools
import time
global_state = 123
if __name__ == "__main__":
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
When I run this program at the command line, I get something like this:
$ python -i main.py
Starting main loop
n=0, m=10, global_state=124
n=1, m=20, global_state=125
n=2, m=30, global_state=126
...
The loop will run for hours or days. Sometimes, while the loop is running, I desire to interactively modify some program state. For example, I'd like to set global_state = -1000 or m = 75. But the loop function is blocking...
Approaches to interactive modification of program state
Approach 1: Stopping and restarting the loop
I can use a KeyboardInterrupt to stop the loop. Since I have invoked python with the -i interactive flag, I can then modify the global state at the interactive REPL. I then restart the loop. This is a simple approach, but it has drawbacks:
- If the loop modifies the global state (or has any side effects), the global state might be left in an inconsistent state when KeyboardInterrupt is raised; the exception could be raised at any point in the loop's control flow, and handling this exception properly can be tricky, especially if the business logic is complex.
- When restarting the loop, the counter
nwill be reset to zero. Other state variables used by the loop (e.g.m) might also be reset or be in an inconsistent state.
Approach 2: Using threading
I can wrap the loop in a function and use threading to run the function in another thread:
def loop():
print("Starting main loop")
m = 0
for n in itertools.count():
time.sleep(1) # "computation"
global_state += 1
m += 10
print(f"{n=}, {m=}, {global_state=}")
import threading
thread = threading.Thread(target=loop)
thread.start()
When I invoke python -i main.py, I can use the REPL to interact with global state as the loop runs in another thread. The loop function still prints to stdout as before. The drawbacks of this approach are:
- I can no longer interact with the loop's local state (e.g. I can't modify
m) because the local state is wrapped in a function. This is a major drawback. - Using threading makes the program more fragile; threads can crash, and exceptions raised in threads need to be handled carefully. If the thread blocks or crashes or hangs, it can be hard to recover.
- Stopping the loop becomes more difficult, because KeyboardInterrupt is no longer an option (instead I need to "cooperate" with the thread, sending it some signal that it should self-terminate).
Approach 3: Drop into REPL from within the loop
I can cooperate with the loop to drop into the repl:
import os
import code
if __name__ == "__main__":
...
for n in itertools.count():
... # business logic
if os.stat("use_repl.txt").st_size > 0: # file nonempty
code.interact(local=locals())
Here we are using a file to signal to the loop when we want to modify state. If the "use_repl.txt" file is nonempty then loop then calls code.interact to create an interactive Console. This has the added benefit of working even if the loop is wrapped in a function; local variables are loaded into the interactive console.
Drawbacks:
- Reading a "use_repl.txt" file at every pass through the loop feels a bit clunky. But how else would one signal (cooperatively) to the loop that it should create a REPL?
- The loop gets paused when
code.interactis called. This differs from thethreading-based solution, where the loop runs continuously. This may be an advantage or a disadvantage, depending on your use-case. - The REPL created by
code.interactis not as polished as python's native interactive mode. For example, tab-completion no longer works.
An alternative to calling code.interact would be to call the breakpoint built-in function. Another alternative would be to invoke the break keyword, exiting the loop. One could then modify state and restart the loop (as in approach 1), without having to worry that a KeyboardInterrupt has created an inconsistent state.
The Question(s):
Are there any approaches that I've missed? Each of the approaches considered has significant drawbacks... Are there any best-practices regarding achievement of the desired result?