In C python, accessing the bytecode evaluation stack

user_546218 0
Viewed 1565

Given a C Python frame pointer, how do I look at arbitrary evaluation stack entries? (Some specific stack entries can be found via locals(), I'm talking about other stack entries.)

I asked a broader question like this a while ago:

getting the C python exec argument string or accessing the evaluation stack

but here I want to focus on being able to read CPython stack entries at runtime.

I'll take a solution that works on CPython 2.7 or any Python later than Python 3.3. However if you have things that work outside of that, share that and, if there is no better solution I'll accept that.

I'd prefer not modifying the C Python code. In Ruby, I have in fact done this to get what I want. I can speak from experience that this is probably not the way we want to work. But again, if there's no better solution, I'll take that. (My understanding wrt to SO points is that I lose it in the bounty either way. So I'm happy go see it go to the person who has shown the most good spirit and willingness to look at this, assuming it works.)

update: See the comment by user2357112 tldr; Basically this is hard-to-impossible to do. (Still, if you think you have the gumption to try, by all means do so.)

So instead, let me narrow the scope to this simpler problem which I think is doable:

Given a python stack frame, like inspect.currentframe(), find the beginning of the evaluation stack. In the C version of the structure, this is f_valuestack. From that we then need a way in Python to read off the Python values/objects from there.

update 2 well the time period for a bounty is over and no one (including my own summary answer) has offered concrete code. I feel this is a good start though and I now understand the situation much more than I had. In the obligatory "describe why you think there should be a bounty" I had listed one of the proffered choices "to draw more attention to this problem" and to that extent where there had been something less than a dozen views of the prior incarnation of the problem, as I type this it has been viewed a little under 190 times. So this is a success. However...

If someone in the future decides to carry this further, contact me and I'll set up another bounty.

Thanks all.

cpython disassembly bytecode reverse-engineering python
4 Answers

I wrote some code to do this. It seems to work so I'll add it to this question.

How it does it is by disassembling the instructions, and using dis.stack_effect to get the effect of each instruction on stack depth. If there's a jump, it sets the stack level at the jump target.

I think stack level is deterministic, i.e. it is always the same at any given bytecode instruction in a piece of code no matter how it was reached. So you can get stack depth at a particular bytecode by looking at the bytecode disassembly directly.

There's a slight catch which is that if you are in an active call, the code position is shown as last instruction being the call, but the stack state is actually that before the call. This is good because it means you can recreate the call arguments from the stack, but you need to be aware that if the instruction is a call that is ongoing, the stack will be at the level of the previous instruction.

Here's the code from my resumable exception thing that does this:

cdef get_stack_pos_after(object code,int target,logger):
    stack_levels={}
    jump_levels={}
    cur_stack=0
    for i in dis.get_instructions(code):
        offset=i.offset
        argval=i.argval
        arg=i.arg
        opcode=i.opcode
        if offset in jump_levels:
            cur_stack=jump_levels[offset]
        no_jump=dis.stack_effect(opcode,arg,jump=False)        
        if opcode in dis.hasjabs or opcode in dis.hasjrel:
            # a jump - mark the stack level at jump target
            yes_jump=dis.stack_effect(opcode,arg,jump=True)        
            if not argval in jump_levels:
                jump_levels[argval]=cur_stack+yes_jump
        cur_stack+=no_jump
        stack_levels[offset]=cur_stack
        logger(offset,i.opname,argval,cur_stack)
    return stack_levels[target]

https://github.com/joemarshall/unthrow

user_153278680

I've tried to do this in this package. As others point out, the main difficulty is in determining the top of the Python stack. I try to do this with some heuristics, which I've documented here.

The overall idea is that by the time my snapshotting function is called, the stack consists of the locals (as you point out), the iterators of nested for loops, and any exception triplets currently being handled. There's enough information in Python 3.6 & 3.7 to recover these states and therefore the stacktop.

I also relied on a tip from user2357112 to pave a way to making this work in Python 3.8.

user_7115850
Related