Thursday, January 17, 2013

Interrupting a Python thread with signals

First off, in Python, as appears to be common knowledge, signals can only be received by the main thread, and usually you need to do synchronization work with Queues or something else to work with other threads, so, what I'll describe below is a different approach to the problem which in effect will give you a way of interrupting a thread in Python from anywhere.

First off, I'll show the code and explain it afterwards:

import time
import sys
import threading

class SigFinish(Exception):
    pass

def throw_signal_function(frame, event, arg):
    raise SigFinish()

def do_nothing_trace_function(frame, event, arg):
    # Note: each function called will actually call this function
    # so, take care, your program will run slower because of that.
    return None

def interrupt_thread(thread):
    for thread_id, frame in sys._current_frames().items():
        if thread_id == thread.ident:  # Note: Python 2.6 onwards
            set_trace_for_frame_and_parents(frame, throw_signal_function)

def set_trace_for_frame_and_parents(frame, trace_func):
    # Note: this only really works if there's a tracing function set in this
    # thread (i.e.: sys.settrace or threading.settrace must have set the
    # function before)
    while frame:
        if frame.f_trace is None:
            frame.f_trace = trace_func
        frame = frame.f_back
    del frame


class MyThread(threading.Thread):

    def run(self):
        # Note: this is important: we have to set the tracing function
        # when the thread is started (we could set threading.settrace
        # before starting this thread to do this externally)
        sys.settrace(do_nothing_trace_function)
        try:
            while True:
                time.sleep(.1)
        except SigFinish:
            sys.stderr.write('Finishing thread cleanly\n')


thread = MyThread()
thread.start()
time.sleep(.5)  # Wait a bit just to see it looping.

interrupt_thread(thread)
sys.stderr.write('Joining\n')
thread.join()  # Joining here: if we didn't interrupt the thread before, we'd be here forever.
sys.stderr.write('Finished\n')



So, there you have it: a hacky approach which will prevent your code from being properly debugged :)

-- note that sys.settrace could be changed for sys.setprofile to achieve the same result -- in which case your program could be debugged but not profiled (which may be a better approach but will make the signal be raised only on function calls, not on line calls).

To sum it up, we use the tracing mechanisms that's intended for debuggers (or profilers) to actually throw the signal for us. This means that we have to enable the tracing on that thread (which will make that thread to execute a bit slower) and set the tracer function to a function which will actually throw the signal.

IMO, it works in a hackish (but nice) way... not sure if I'd use that in a production environment, but, there you have it: interruptible threads in Python (yes, you still have the limitation of the GIL: a thread won't be interrupted while calling some atomic operation that doesn't release the GIL)... Now, if only there was a Python signal library better integrated with the Python interpreter so that it checked for signals itself (probably in a way close to the tracing function itself with less overhead as only a single additional check for a null variable would be needed -- and otherwise a signal would be thrown), this hack wouldn't be needed in the first place -- but I'll leave that to someone else reading this :)

Monday, January 07, 2013

Python to Javascript translation

I'm currently doing a project with a backend in Python and a client mostly in Javascript -- still not public, but if everything turns out right I'll post about it later on -- and there are some algorithms I'd like to reuse in both, but without going to a full Python in browser style (such as Pyjs or brython)... Ideally I'd like something like Coffescript (but with a Python-like syntax so that a subset of Python can be used for both Python and Javascript generation).

-- Sidenote: I decided I really have to grasp Javascript in order to find out how things work before trying an alternative language which compiles down to Javascript -- so far my experience has been nice. I found that I end up programming much more in a functional style and use classes in a very limited way (haven't felt the need for inheritance so far), although I feel that it's hard to find resources on what would be the proper way of doing things (but in the plus side, there are LOTS of javascript-related resources, so, I've hardly found myself stuck while learning it).

Anyways, I have some algorithms I've written in Python which I'd like to reuse in Javascript without having to use huge runtime with it (I'm restricting that to simple functions) and searching for a translator, I've found Pyvascript, which is a Python-like language which compiles to Javascript (so, there's only a subset of Python that can really be used, which is ok as I'm not after the ones that attempt to do full Python in Javascript and end up with a huge emulated runtime), but it does fit my needs properly with some minor changes (and now I have some code I can use in both Python and Javascript at the same time). Just to note, it doesn't seem like the project is active right now, but it does work fine for me :)

Wednesday, January 02, 2013

Python: get parent process id (pid) in windows

Below is code to monkey-patch the os module to provide a getppid() function to get the parent process id in windows using ctypes (note that on Python 3.2, os.getppid() already works and is available on windows, but if you're on an older version, this can be used as a workaround).

import os
if not hasattr(os, 'getppid'):
    import ctypes

    TH32CS_SNAPPROCESS = 0x02L
    CreateToolhelp32Snapshot = ctypes.windll.kernel32.CreateToolhelp32Snapshot
    GetCurrentProcessId = ctypes.windll.kernel32.GetCurrentProcessId

    MAX_PATH = 260

    _kernel32dll = ctypes.windll.Kernel32
    CloseHandle = _kernel32dll.CloseHandle

    class PROCESSENTRY32(ctypes.Structure):
        _fields_ = [
            ("dwSize", ctypes.c_ulong),
            ("cntUsage", ctypes.c_ulong),
            ("th32ProcessID", ctypes.c_ulong),
            ("th32DefaultHeapID", ctypes.c_int),
            ("th32ModuleID", ctypes.c_ulong),
            ("cntThreads", ctypes.c_ulong),
            ("th32ParentProcessID", ctypes.c_ulong),
            ("pcPriClassBase", ctypes.c_long),
            ("dwFlags", ctypes.c_ulong),

            ("szExeFile", ctypes.c_wchar * MAX_PATH)
        ]

    Process32First = _kernel32dll.Process32FirstW
    Process32Next = _kernel32dll.Process32NextW

    def getppid():
        '''
        :return: The pid of the parent of this process.
        '''
        pe = PROCESSENTRY32()
        pe.dwSize = ctypes.sizeof(PROCESSENTRY32)
        mypid = GetCurrentProcessId()
        snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0)

        result = 0
        try:
            have_record = Process32First(snapshot, ctypes.byref(pe))

            while have_record:
                if mypid == pe.th32ProcessID:
                    result = pe.th32ParentProcessID
                    break

                have_record = Process32Next(snapshot, ctypes.byref(pe))

        finally:
            CloseHandle(snapshot)

        return result

    os.getppid = getppid