Showing the stack trace from a running Python application

I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons. Its a bit hacky, and only works on unix (requires signals):

import code, traceback, signal

def debug(sig, frame):
    """Interrupt running process, and provide a python prompt for
    interactive debugging."""
    d={'_frame':frame}         # Allow access to frame object.
    d.update(frame.f_globals)  # Unless shadowed by global
    d.update(frame.f_locals)

    i = code.InteractiveConsole(d)
    message  = "Signal received : entering python shell.\nTraceback:\n"
    message += ''.join(traceback.format_stack(frame))
    i.interact(message)

def listen():
    signal.signal(signal.SIGUSR1, debug)  # Register handler

To use, just call the listen() function at some point when your program starts up (You could even stick it in site.py to have all python programs use it), and let it run. At any point, send the process a SIGUSR1 signal, using kill, or in python:

    os.kill(pid, signal.SIGUSR1)

This will cause the program to break to a python console at the point it is currently at, showing you the stack trace, and letting you manipulate the variables. Use control-d (EOF) to continue running (though note that you will probably interrupt any I/O etc at the point you signal, so it isn't fully non-intrusive.

I've another script that does the same thing, except it communicates with the running process through a pipe (to allow for debugging backgrounded processes etc). Its a bit large to post here, but I've added it as a python cookbook recipe.

The suggestion to install a signal handler is a good one, and I use it a lot. For example, bzr by default installs a SIGQUIT handler that invokes pdb.set_trace() to immediately drop you into a pdb prompt. (See the bzrlib.breakin module's source for the exact details.) With pdb you can not only get the current stack trace (with the (w)here command) but also inspect variables, etc.

However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in ~/.gdbinit, then:

• Attach gdb: gdb -p PID
• Get the python stack trace: pystack

It's not totally reliable unfortunately, but it works most of the time. See also https://wiki.python.org/moin/DebuggingWithGdb

Finally, attaching strace can often give you a good idea what a process is doing.

>>> import traceback
>>> def x():
>>>    print traceback.extract_stack()

>>> x()
[('<stdin>', 1, '<module>', None), ('<stdin>', 2, 'x', None)]

You can also nicely format the stack trace, see the docs.

Edit: To simulate Java's behavior, as suggested by @Douglas Leeder, add this:

import signal
import traceback

signal.signal(signal.SIGUSR1, lambda sig, stack: traceback.print_stack(stack))

to the startup code in your application. Then you can print the stack by sending SIGUSR1 to the running Python process.

The traceback module has some nice functions, among them: print_stack:

import traceback

traceback.print_stack()

What really helped me here is spiv's tip (which I would vote up and comment on if I had the reputation points) for getting a stack trace out of an unprepared Python process. Except it didn't work until I modified the gdbinit script. So:

• download https://svn.python.org/projects/python/trunk/Misc/gdbinit and put it in ~/.gdbinit

• edit it, changing PyEval_EvalFrame to PyEval_EvalFrameEx [edit: no longer needed; the linked file already has this change as of 2010-01-14]

• Attach gdb: gdb -p PID

• Get the python stack trace: pystack

python -dv yourscript.py

That will make the interpreter to run in debug mode and to give you a trace of what the interpreter is doing.

If you want to interactively debug the code you should run it like this:

python -m pdb yourscript.py

That tells the python interpreter to run your script with the module "pdb" which is the python debugger, if you run it like that the interpreter will be executed in interactive mode, much like GDB

It can be done with excellent py-spy. It's a sampling profiler for Python programs, so its job is to attach to a Python processes and sample their call stacks. Hence, py-spy dump --pid $SOME_PID is all you need to do to dump call stacks of all threads in the $SOME_PID process. Typically it needs escalated privileges (to read the target process' memory).

Here's an example of how it looks like for a threaded Python application.

$ sudo py-spy dump --pid 31080
Process 31080: python3.7 -m chronologer -e production serve -u www-data -m
Python v3.7.1 (/usr/local/bin/python3.7)

Thread 0x7FEF5E410400 (active): "MainThread"
    _wait (cherrypy/process/wspbus.py:370)
    wait (cherrypy/process/wspbus.py:384)
    block (cherrypy/process/wspbus.py:321)
    start (cherrypy/daemon.py:72)
    serve (chronologer/cli.py:27)
    main (chronologer/cli.py:84)
    <module> (chronologer/__main__.py:5)
    _run_code (runpy.py:85)
    _run_module_as_main (runpy.py:193)
Thread 0x7FEF55636700 (active): "_TimeoutMonitor"
    run (cherrypy/process/plugins.py:518)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)
Thread 0x7FEF54B35700 (active): "HTTPServer Thread-2"
    accept (socket.py:212)
    tick (cherrypy/wsgiserver/__init__.py:2075)
    start (cherrypy/wsgiserver/__init__.py:2021)
    _start_http_thread (cherrypy/process/servers.py:217)
    run (threading.py:865)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)
...
Thread 0x7FEF2BFFF700 (idle): "CP Server Thread-10"
    wait (threading.py:296)
    get (queue.py:170)
    run (cherrypy/wsgiserver/__init__.py:1586)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)  

I was looking for a while for a solution to debug my threads and I found it here thanks to haridsv. I use slightly simplified version employing the traceback.print_stack():

import sys, traceback, signal
import threading
import os

def dumpstacks(signal, frame):
  id2name = dict((th.ident, th.name) for th in threading.enumerate())
  for threadId, stack in sys._current_frames().items():
    print(id2name[threadId])
    traceback.print_stack(f=stack)

signal.signal(signal.SIGQUIT, dumpstacks)

os.killpg(os.getpgid(0), signal.SIGQUIT)

For my needs I also filter threads by name.

I hacked together some tool which attaches into a running Python process and injects some code to get a Python shell.

See here: https://github.com/albertz/pydbattach

It's worth looking at Pydb, "an expanded version of the Python debugger loosely based on the gdb command set". It includes signal managers which can take care of starting the debugger when a specified signal is sent.

A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb.

Since Austin 3.3, you can use the -w/--where option to emit the current stack trace. See https://stackoverflow.com/a/70905181/1838793

If you want to peek at a running Python application to see the "live" call stack in a top-like fashon you can use austin-tui (https://github.com/p403n1x87/austin-tui). You can install it from PyPI with e.g.

pipx install austin-tui

Note that it requires the austin binary to work (https://github.com/p403n1x87/austin), but then you can attach to a running Python process with

austin-tui -p <pid>

How to debug any function in console:

Create function where you use pdb.set_trace(), then function you want debug.

>>> import pdb
>>> import my_function

>>> def f():
...     pdb.set_trace()
...     my_function()
... 

Then call created function:

>>> f()
> <stdin>(3)f()
(Pdb) s
--Call--
> <stdin>(1)my_function()
(Pdb) 

Happy debugging :)

You can use the hypno package, like so:

hypno <pid> "import traceback; traceback.print_stack()"

This would print a stack trace into the program's stdout.

Alternatively, if you don't want to print anything to stdout, or you don't have access to it (a daemon for example), you could use the madbg package, which is a python debugger that allows you to attach to a running python program and debug it in your current terminal. It is similar to pyrasite and pyringe, but newer, doesn't require gdb, and uses IPython for the debugger (which means colors and autocomplete).

To see the stack trace of a running program, you could run:

madbg attach <pid>

And in the debugger shell, enter: bt

Disclaimer - I wrote both packages

I don't know of anything similar to java's response to SIGQUIT, so you might have to build it in to your application. Maybe you could make a server in another thread that can get a stacktrace on response to a message of some kind?

There is no way to hook into a running python process and get reasonable results. What I do if processes lock up is hooking strace in and trying to figure out what exactly is happening.

Unfortunately often strace is the observer that "fixes" race conditions so that the output is useless there too.

At the point where the code is run, you can insert this small snippet to see a nicely formatted printed stack trace. It assumes that you have a folder called logs at your project's root directory.

# DEBUG: START DEBUG -->
import traceback

with open('logs/stack-trace.log', 'w') as file:
    traceback.print_stack(file=file)
# DEBUG: END DEBUG --!