In Python, is there any way to save/dump/serialize a PDB debugging session?

user_1473169 0
Viewed 318

I've been working with Python programs which take several hours to complete, but crash occasionally. To debug, so far I have been adding conditional breakpoints, which drop me into a PDB session whenever a problem occurs. This is great because pinpointing the exact cause of the problem is hard, and the interactive session lets me explore the whole program (including all the stack frames and so on).

The only issue is, if I ever accidentally close or crash my debugging session, I need to start the whole program again! Reaching my breakpoint takes several hours! I would really, really like a way of serializing a PDB session and re-opening it multiple times. Does anything like this exist? I have looked into dill to serialize an interpreter session, unfortunately several of my types fail to serialize (it also isn't robust to code changes down the line). Thanks!

pdb dump serialization python debugging
1 Answers

You haven't specified your operating system of choice, but in linux world there is a criu utility - https://criu.org/Main_Page , which can be used to save an application state. Now there are a lot of pitfalls, especially with tty-based applications (see https://criu.org/Advanced_usage#Shell_jobs_C.2FR) but here is an exaple.

  1. I got a simple python application with pdb debug point, let's call it app.py:

    print("hello")
import pdb; pdb.set_trace()
print("world")

  2. After running this application with python app.py you get expected

    hello
> /home/user/app.py(3)<module>()
-> print("world")

  3. Get your pid with pgrep -f app.py, in my case it was 17060

  4. Create a folder to dump your process

    mkdir /tmp/criu

  5. Dump your process with

    sudo criu dump -D /tmp/criu -t 17060 --shell-job

    notice that you current process will be killed (AFAIK due to --shell-job key, see link above). you'll see

    (Pdb) [1]    17060 killed     python app.py

    in your tty

  6. Restore your process with

    sudo criu restore -D /tmp/criu --shell-job

    your tty will be restored in a same window where you used this command.

  7. Since debugger is attached, you may type c and enter to see if it actually worked. Here's the result on my machine:

    (Pdb) c
world

Hope that helps, there are a lot of pitfalls that might make this approach unfeasible for you.

The other way is to run your code in a VM and snapshot disk and memory each time. It might not be the best solution resource-wise, but many hypervisors have a nice UI, or even shell utilities to control state of virtual machines. Snapshoting tech is mature in any hypervisor now days, you shouldn't run into any problems. Setup remote debugging and connect with your favorite IDE after bringing your snapshot back.

Edit: There is also an easy way to do this if you are running your applications in containers and you OS supports podman and criu 3.11+

https://criu.org/Podman

You can use something like

podman run -d --name your_container_name your_image

To snapshot use

podman container checkpoint your_container_id

To restore use

podman container restore your_container_id

All these commands require root priviliges. Unfortunately i wasn't able to test it because my distro provides criu 3.8, and for podman 3.11 is required.

Same functionality is available as experimental flag in Docker, see https://criu.org/Docker

user_30660810
Related