Tuesday, February 04, 2014

Changing the locals of a frame (frame.f_locals) and persisting results (with ctypes)

Up until now I didn't know of a proper way to change the locals of a frame (out of the normal execution flow in Python), so, when in the PyDev debugger changing a variable wouldn't always work.

So, for instance, if you have a frame (which you could get from a traceback, sys._getframe().f_back, etc), you could get its locals with frame.f_locals, but changing the frame.f_locals (which gives you a dictionary) wouldn't apply the results back to the frame.

This is mostly due to how CPython works: frame.f_locals actually creates a dictionary using PyFrame_FastToLocals, but changes to the dictionary aren't applied back.

Some years ago I had found a way to make it work (see: http://bugs.python.org/issue1654367) through a CPython function: PyFrame_FastToLocals, but up until recently, I thought it needed a modified version of CPython in order to work, now, recently I discovered ctypes can access a lot from the python api (through ctypes.pythonapi):

So, after changing frame.f_locals, it's possible to use ctypes to call PyFrame_LocalsToFast doing:


import ctypes

ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))

 

A note: the second parameter (which may be 0 or 1) defines whether we want to erase variables removed from the dict (which would require 1) or not.

So,  the PyDev debugger now incorporates this utility so that if you're running in CPython, it will properly change the variable in a scope when you change a variable :)

Note that this isn't compatible with other Python implementations (this is not something the language dictates how it should work -- probably the ideal would be making frame.f_locals writable).


3 comments:

  1. kousu1:35 AM

    Thank you for posting this! It's half a decade later but I was able to use your information to build a better breakpoint() hook for modern python.

    The key turned out to be calling LocalsToFast() before *each* update to a variable:

    ```
    import ctypes
    ctypes.pythonapi.PyFrame_LocalsToFast(
    ctypes.py_object(frame),
    ctypes.c_int(1))
    for k, v in f_locals.items():
    frame.f_locals[k] = v
    ctypes.pythonapi.PyFrame_LocalsToFast(
    ctypes.py_object(frame),
    ctypes.c_int(1))
    ```

    I found that doing it in the other order gave bizarre results. For example, if in my frame `a=1; b=2; c=3` then

    ```
    frame.f_locals['a'] = 17

    import ctypes
    ctypes.pythonapi.PyFrame_LocalsToFast(
    ctypes.py_object(frame),
    ctypes.c_int(1))
    frame.f_locals['b'] = 29

    import ctypes
    ctypes.pythonapi.PyFrame_LocalsToFast(
    ctypes.py_object(frame),
    ctypes.c_int(1))
    frame.f_locals['c'] = 84
    ```

    would produce `a=17; b=2; c=3`

    ReplyDelete
  2. Hi @kousu, if you want you can take a look at: https://github.com/fabioz/PyDev.Debugger/blob/master/_pydevd_bundle/pydevd_save_locals.py (after some discussions on this, Pypy added support for that too but under a different API and that code also handles Pypy).

    ReplyDelete
  3. Thank you! I used this snippet to update generators on-the-fly in this little side project: https://github.com/alfiopuglisi/pipeline

    ReplyDelete