PyDev funding and LiClipse?

I got some feedback related to the creation of LiClipse and its relation with PyDev in the current crowdfunding proposal (http://igg.me/at/liclipse/), so, I'd like to explain how I believe things fit together.

1. So, is LiClipse a fork of Eclipse?

Definitely not. The idea behind LiClipse is having 2 things:

• A way to theme Eclipse itself better and have some other UI improvements (I'm truly annoyed by not having what I'd consider a professional dark theme in Eclipse right now, so, I'd like to take those matters in my own hands).
• An editor which should be able to support lots of languages out of the box. Think something closer to other all-purpose editors -- as opposed to IDEs -- such as Notepad++, Vi, TextMate, Sublime, etc. i.e.: the idea is supporting lots of languages out of the box, so, the idea is having it resembling formats such as ultraedit wordfiles or TextMate language files.

To be clear, the idea is not to replace the more advanced editors inside Eclipse for each language, but to provide a lightweight way to deal with any language -- usually you work with 1 or 2 main languages, for which you'll have the plugins you need, but sometimes, when you just want to open a file in a language  you work seldomly and may not need/want to install a bulkier plugin, LiClipse would be a good addition to your toolbox (LiClipse is a short for "Lightweight Eclipse" BTW), and for me, not having this is a major shortcoming of Eclipse itself (and it may be an alternative for people who want less features and more speed).

Also, that should be doable without having to create a fork of Eclipse (although some theming issues may need to be resolved at Eclipse itself -- but on those parts, things should be fixed at that level, not on LiClipse).

2. How does that relate to PyDev?

Well, PyDev is very tightly bound to the environment it works on (Eclipse), and I'd like to solve some of the issues I see in it to improve its ecosystem as a whole, which IMHO is something I see needed for PyDev and Eclipse itself to keep moving forward.

3. But won't that divert too many resources out of PyDev itself?

For this proposal, my plan is spending my time 50/50 (I don't think LiClipse is a major undertaking -- all the pieces are out there, it's mostly a matter of stitching them together), although during that time, yes, it'll divert some resources from PyDev, but as I think that having it is very important for PyDev itself (as well as other language), I see the issues being tackled as very serious shortcomings of Eclipse which hinder the adoption of Eclipse itself (thus affecting PyDev directly).

4. I still think LiClipse is not a good idea and would like to support only PyDev.

Please e-mail me with those thoughts. I really believe LiClipse is needed for PyDev to keep getting traction, but if you feel that's not the case, please, please share your thoughts with me.

Also, just to make things clear, if the funding at http://igg.me/at/liclipse/ does not succeed, I won't really be able to support PyDev itself anymore (personally, I really want it to succeed, but I see the funding as the community speaking, so, if after 10 years working on it the PyDev community doesn't see it as a worthy goal or doesn't trust me enough to support me on LiClipse while properly maintaining PyDev, well, I really need to hear it and move on -- as a note, until now, I see the funding as a huge success, getting to 10% in the first day, so, if everyone keeps helping a bit there, it'll be awesome :)

Keeping PyDev alive through crowdfunding

Ok, I just started a crowdfunding project (at www.indiegogo.com) for the continued development of PyDev (and improvements on Eclipse overall).

In this post, I hope to shed some light on why this funding is needed.

First, a little bit of history just to give some context: I've been developing PyDev for more than 10 years already (wow, I just realized that as I started the campaign) and it's definitely a pretty successful project (if it's not the most used Python IDE, it's definitely among the top ones).

So, backtrack a few years...

PyDev started doing success and I created a commercial extension to enable me to work more time on PyDev itself. After some time, Aptana acquired it and I joined them. Unfortunately, Aptana itself didn't turn out very well: it focused on its main product (the IDE) without generating any revenue from it, while trying to make side projects worthy enough to cover for everything (but those side projects weren't successful enough for that).

Enter Appcelerator: it acquired Aptana out of the need to provide a deeper integration for its main product (which is Titanium). So, for some time, it did back up the PyDev development (along with the other languages), but in the end, their main product is Titanium and the tools around it, so, PyDev itself wasn't seen as relevant enough to be kept supported (to be clear, they're still hosting the homepage and downloads, but not backing the development itself). As such, in the end of the last year they stopped supporting its development.

So, that's where the project is at now: it's (IMHO) a pretty successful, but unable to generate revenue for its continued support. Given this scenario I decided to create a crowdfunding project to ask for the community to provide resources to make that happen.

In the funding, I expanded its reach a bit, on what I think are the main issues with the PyDev environment right now. So, the idea is focusing on a nicer dark UI, usability, speed and memory and providing a way to easily have other editors out of the box in a lightweight implementation (Python is pretty strong in the web, so, one of the weak points right now is actually not the Python editing itself, but related web languages, such as CoffeScript and JavaScript).

So, please help in funding (and sharing) at http://igg.me/at/liclipse to keep PyDev going strong!

Interrupting a Python thread with signals

First off, in Python, as appears to be common knowledge, signals can only be received by the main thread, and usually you need to do synchronization work with Queues or something else to work with other threads, so, what I'll describe below is a different approach to the problem which in effect will give you a way of interrupting a thread in Python from anywhere.

First off, I'll show the code and explain it afterwards:

import time
import sys
import threading

class SigFinish(Exception):
    pass

def throw_signal_function(frame, event, arg):
    raise SigFinish()

def do_nothing_trace_function(frame, event, arg):
    # Note: each function called will actually call this function
    # so, take care, your program will run slower because of that.
    return None

def interrupt_thread(thread):
    for thread_id, frame in sys._current_frames().items():
        if thread_id == thread.ident:  # Note: Python 2.6 onwards
            set_trace_for_frame_and_parents(frame, throw_signal_function)

def set_trace_for_frame_and_parents(frame, trace_func):
    # Note: this only really works if there's a tracing function set in this
    # thread (i.e.: sys.settrace or threading.settrace must have set the
    # function before)
    while frame:
        if frame.f_trace is None:
            frame.f_trace = trace_func
        frame = frame.f_back
    del frame


class MyThread(threading.Thread):

    def run(self):
        # Note: this is important: we have to set the tracing function
        # when the thread is started (we could set threading.settrace
        # before starting this thread to do this externally)
        sys.settrace(do_nothing_trace_function)
        try:
            while True:
                time.sleep(.1)
        except SigFinish:
            sys.stderr.write('Finishing thread cleanly\n')


thread = MyThread()
thread.start()
time.sleep(.5)  # Wait a bit just to see it looping.

interrupt_thread(thread)
sys.stderr.write('Joining\n')
thread.join()  # Joining here: if we didn't interrupt the thread before, we'd be here forever.
sys.stderr.write('Finished\n')

So, there you have it: a hacky approach which will prevent your code from being properly debugged :)

-- note that sys.settrace could be changed for sys.setprofile to achieve the same result -- in which case your program could be debugged but not profiled (which may be a better approach but will make the signal be raised only on function calls, not on line calls).

To sum it up, we use the tracing mechanisms that's intended for debuggers (or profilers) to actually throw the signal for us. This means that we have to enable the tracing on that thread (which will make that thread to execute a bit slower) and set the tracer function to a function which will actually throw the signal.

IMO, it works in a hackish (but nice) way... not sure if I'd use that in a production environment, but, there you have it: interruptible threads in Python (yes, you still have the limitation of the GIL: a thread won't be interrupted while calling some atomic operation that doesn't release the GIL)... Now, if only there was a Python signal library better integrated with the Python interpreter so that it checked for signals itself (probably in a way close to the tracing function itself with less overhead as only a single additional check for a null variable would be needed -- and otherwise a signal would be thrown), this hack wouldn't be needed in the first place -- but I'll leave that to someone else reading this :)

Python to Javascript translation

I'm currently doing a project with a backend in Python and a client mostly in Javascript -- still not public, but if everything turns out right I'll post about it later on -- and there are some algorithms I'd like to reuse in both, but without going to a full Python in browser style (such as Pyjs or brython)... Ideally I'd like something like Coffescript (but with a Python-like syntax so that a subset of Python can be used for both Python and Javascript generation).

-- Sidenote: I decided I really have to grasp Javascript in order to find out how things work before trying an alternative language which compiles down to Javascript -- so far my experience has been nice. I found that I end up programming much more in a functional style and use classes in a very limited way (haven't felt the need for inheritance so far), although I feel that it's hard to find resources on what would be the proper way of doing things (but in the plus side, there are LOTS of javascript-related resources, so, I've hardly found myself stuck while learning it).

Anyways, I have some algorithms I've written in Python which I'd like to reuse in Javascript without having to use huge runtime with it (I'm restricting that to simple functions) and searching for a translator, I've found Pyvascript, which is a Python-like language which compiles to Javascript (so, there's only a subset of Python that can really be used, which is ok as I'm not after the ones that attempt to do full Python in Javascript and end up with a huge emulated runtime), but it does fit my needs properly with some minor changes (and now I have some code I can use in both Python and Javascript at the same time). Just to note, it doesn't seem like the project is active right now, but it does work fine for me :)

Python: get parent process id (pid) in windows

Below is code to monkey-patch the os module to provide a getppid() function to get the parent process id in windows using ctypes (note that on Python 3.2, os.getppid() already works and is available on windows, but if you're on an older version, this can be used as a workaround).

import os
if not hasattr(os, 'getppid'):
    import ctypes

    TH32CS_SNAPPROCESS = 0x02L
    CreateToolhelp32Snapshot = ctypes.windll.kernel32.CreateToolhelp32Snapshot
    GetCurrentProcessId = ctypes.windll.kernel32.GetCurrentProcessId

    MAX_PATH = 260

    _kernel32dll = ctypes.windll.Kernel32
    CloseHandle = _kernel32dll.CloseHandle

    class PROCESSENTRY32(ctypes.Structure):
        _fields_ = [
            ("dwSize", ctypes.c_ulong),
            ("cntUsage", ctypes.c_ulong),
            ("th32ProcessID", ctypes.c_ulong),
            ("th32DefaultHeapID", ctypes.c_int),
            ("th32ModuleID", ctypes.c_ulong),
            ("cntThreads", ctypes.c_ulong),
            ("th32ParentProcessID", ctypes.c_ulong),
            ("pcPriClassBase", ctypes.c_long),
            ("dwFlags", ctypes.c_ulong),

            ("szExeFile", ctypes.c_wchar * MAX_PATH)
        ]

    Process32First = _kernel32dll.Process32FirstW
    Process32Next = _kernel32dll.Process32NextW

    def getppid():
        '''
        :return: The pid of the parent of this process.
        '''
        pe = PROCESSENTRY32()
        pe.dwSize = ctypes.sizeof(PROCESSENTRY32)
        mypid = GetCurrentProcessId()
        snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0)

        result = 0
        try:
            have_record = Process32First(snapshot, ctypes.byref(pe))

            while have_record:
                if mypid == pe.th32ProcessID:
                    result = pe.th32ParentProcessID
                    break

                have_record = Process32Next(snapshot, ctypes.byref(pe))

        finally:
            CloseHandle(snapshot)

        return result

    os.getppid = getppid