Thursday, September 25, 2014

Attaching debugger to running process (PyDev 3.8.0)

The latest PyDev 3.8.0 has just been released... along with a bunch of bugfixes, the major feature added is the possibility of attaching the debugger to a running process.

So, I thought about explaining a bit how to use it (and later a bit on how it was done).

The first thing is that the Debug perspective must be activated (as the attach to process menu is only shown by default in the Debug Perspective). Then, when on the debug perspective, select PyDev > Attach to Process (as the image below shows).

When that action is activated, a list with the active processes is shown so that the process we should attach to can be selected (as a note, by default only Python processes are filtered, but if you have an executable running Python under a different name, it's also possible to select it).

After selecting the executable, the user must provide a Python interpreter that's compatible with the interpreter we'll attach to (i.e.: if we're attaching to a 64 bit interpreter, the interpreter selected must also be a 64 bit process -- and likewise for 32 bits).

And if everything went right, we'll be connected with the process after that step (you can note in the screenshot below that in the Debug View we have a process running regularly and we connected to the process through the remote debugger) -- after it's connected, it's possible to pause the running process (through right-click process > suspend)

So, this was how to use it within PyDev... now, I'll explain a bit on the internals as I had a great time implementing that feature :)

The first thing to note is that we currently support (only) Windows and Linux... although the basic idea is the same on both cases (we get a dll loaded in the target process and then execute our attach code through it), the implementations are actually pretty different (as it's Open Source, it can be seen at:

So, let me start with the Windows implementation...

In the Windows world, I must thank a bunch of people that came before me in order to make it work:
First, Mario Vilas for Winappdbg: -- mostly, this is a Windows debugger written in Python. We use it to attach a dll to a target process. Then, after the dll is loaded, there's some hand-crafted shellcode created to execute a function from that dll (which is actually different for 32 bits and for 64 bits -- I did spend quite some time here since my assembly knowledge was mostly theoretical, so, it was nice to see it working in practice: that's the GenShellCodeHelper class in and it's used in the run_python_code_windows function).

Ok, so, that gives us the basic hackery needed in order to execute some code in a different process (which Winappdbg does through the Windows CreateRemoteThread API) -- so, initially I had a simple version working which did (all in shellcode) an acquire of the Python GIL/run some hand-crafted Python code through PyRun_SimpleString/release GIL... but there are a number of problems with that simple approach: the first one is that although we'll execute some Python code there, we'll do it under a new thread, which under Python 3 meant that this would be no good since we have to initialize the Python threading if it still wasn't initialized. Also, to setup the Debugger we need to call sys.settrace on existing threads (while executing in that thread -- or at least making Python think we're at that thread as there's no API for that)... at that point I decided on having the dll (and not only shellcode).

So, here I must thank the PVTS guys (which actually have an attach to process on Windows which does mixed mode debugging), so, the attach.cpp file which generates our dll is adapted from PVTS to the PyDev use case. It goes through a number of hoops to initialize the threading facility in Python if it still wasn't initialized and does the sys.settrace while having all threads suspended and makes Python think we're actually at the proper thread to make that call... looking at it, I find it really strange that Python itself doesn't have an API for that (it should be easy to do on Python, but it's such a major hack on the debugger because that API is not available).

Now, on to Linux: in Linux the approach is simpler as we reuse a debugger that should be readily available for Linux developers: gdb. Also, because gdb stops threads for us and executes the code in an existing thread, things become much simpler... first, because we're executing in an existing thread, we don't have to start the threading if it's not started -- the only reason this is needed in Windows is because we're executing the code in a new thread in the process, created through CreateRemoteThread -- and also, as gdb has a way to script in Python were we can switch threads and execute something in it while having other threads stopped, we also don't need to do the trick on Python to make it think we're in a thread to execute a sys.settrace as if we were in a thread when in reality we weren't, as we can switch to a thread and really execute code on it.

So, all in all, it should be working properly, although there are a number of caveats... it may fail if we don't have permissions for the CreateRemoteThread on Windows or in Linux it could fail if the ptrace permissions are not set for us to attach with GDB -- and probably a bunch of other things I still didn't think of :)

Still, it's nice to see it working!

Also, the last thanks goes to JetBrains/PyCharm, which helped in sponsoring the work in the debugger -- as I mentioned earlier: the debugger in PyDev/PyCharm is now merged :)


Tolerious Feng said...


Anonymous said...

Great work, Fabio, and thanks for explaining!

Alex Waters said...

Any chance this will be available to OSX? Is it already?

Fabio Zadrozny said...

Hi Alex,

It's not currently available in OSX... the main issue is that the dlls must be compiled on OSX (the main issue being that I don't have a Mac).

I believe that the needed point would be compiling the Unix dll to Mac OS:

then making the run_python_code_linux aware that it could be used in mac os at (to properly locate the mac os dll).

and testing/distributing the dll.

If anyone would be willing to do that, it'd be nice.

As a note, I do have plans on getting a Mac myself to handle those things better -- contributions to PyDev should allow me to do that somewhere along the short-medium term -- it's just not there yet.

Best Regards,


Alex Waters said...

Do you accept Bitcoin? If so, what is the project's bitcoin address? If not, ;)

Going to buy a license anyway, I just prefer to send btc.

Fabio Zadrozny said...

Hi Alex, thanks for the tip, will take a look at coin co.

Currently, I actually have a Bitcoin address, but I don't have anything integrated to send invoice/licenses when I receive by it, so, anyone can do an anonymous donation through it (see address at the end of, but an invoice/license won't be received in this case (so, asked at coin co to check if my scenario can be dealt with properly).

Debarshi said...

Thank you for this information.
I want to attach an process which embeds a Python interpreter, by calling python dll APIs directly. When I tried to attach the process the debugger fails with below message. Please help me.

Connecting to 64 bits target
Traceback (most recent call last):
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\", line 64, in
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\", line 61, in main
setup['pid'], python_code, connect_debugger_tracing=True, show_debug_info=show_debug_info_on_target_process)
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\", line 283, in run_python_code_windows
assert resolve_label(process, compat.b('PyGILState_Ensure'))
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\", line 251, in resolve_label
address = process.resolve_label(label)
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\winappdbg\", line 1546, in resolve_label
address = self.resolve_label_components(module, function, offset)
File "D:\Installs_Tools\eclipse\plugins\org.python.pydev_3.9.0.201411111611\pysrc\pydevd_attach_to_process\winappdbg\", line 1635, in resolve_label_components
raise RuntimeError(msg)
RuntimeError: Function b'PyGILState_Ensure' not found in any module
Process finished with exitValue: 1

Fabio Zadrozny said...

Hi there,

Well, I'm not sure I can help very well when you have it embedded in another process (it does a lot expecting it to be python) -- i.e.: in this case, it's not able to find the PyGILState_Ensure symbol in your process (which would usually be found -- and there are many other symbols it'll expect to find).

So, if you don't link the Python DLL in your program, I don't really see a way for that attach to work...

Now, maybe what you want can be solved by using the remote debugger API? (mostly calling pydevd.settrace() to connect -- after starting the remote debugger on PyDev... see for details).

Debarshi said...

Thank you Fabio. I am able to debug now.