Mastering writing code on PyDev

One of these days I was pair coding with a friend -- which was also using PyDev -- and upon looking him code I realized that there are some simple things which everyone should know when writing code on PyDev:

1. Want to write a new line? Use Shift+Enter: This is what I almost always use in PyDev instead of the plain enter to enter a new line.

How it works: It'll emulate moving to the end of the line and pressing enter there.

Tip: It can be used when there's a completion to avoid applying it while going to the next line (as Enter is also used to apply a completion if you don't use Shift+Enter you have to press ESC before to avoid applying it if you use a plain Enter).

2. Want to copy or move some lines? Ctrl+Alt+Up will copy the lines up and Ctrl+Alt+Down will copy them down and then Alt+Up or Alt+Down can be used to move them up or down (so, it's a line-oriented copy/paste which is really handy).

3. Is there a string or comment which is too big and you want to wrap it? Use Ctrl+2, W and it'll wrap it for you (using the number of columns assigned in the print margin preferences).

4. Want to create a docstring for the parameters of a method? Ctrl+1 -> Make Docstring (in a line with the method declaration).

5. Assign parameters to variables? Ctrl+1 -> Assign parameters to attributes (in a line with the method declaration in a class). Alternatively use Ctrl+2, A.

6. Want to select the current word? Shift+Alt+Up will select the current word for you (and if you keep using Shift+Alt+Up it'll then select the enclosing context and Shift+Alt+Down will deselect it again).

7. Rectangular selection: Shift+Alt+A.

8. Want to rename a local token? To do that use Ctrl+2, R (note that for renaming a token in multiple modules you'd use Shift+Alt+R or a package/module in the PyDev Package Explorer would be renamed with F2).

9. To rename something that's not a token (such as a comma) -- if you're in LiClipse (which adds this feature) -- you can do Ctrl+K (or Ctrl+Shift+K to go backwards) multiple times to mark the places you want to rename and then go on to rename it (as a note, Ctrl+Alt+K will unlink one of the occurrences). This can also be used for a substring -- or any character combination for that matter.

10. Do you have some code with multiple tokens that are not found and need to be imported? Ctrl+Shift+O can be used to resolve the multiple missing tokens. Alternatively you could also use Ctrl+1 on a line with a missing import to get the suggestion to add it (if it wasn't already added in a code-completion which added it for you automatically).

11. Like TDD coding? i.e.: write the test first and do the code later. If so, you can usually write the test and use Ctrl+1 to provide a suggestion to create the missing method, module or attribute.

12. PyDev doesn't automatically re-analyze dependent modules when one module changes, so, I've seen people just add some space and save the file again just to force a new code analysis... while this works, there's a specific keybinding: Ctrl+2, C which will force a code-analysis in the current module.

13. When there are many contiguous lines commented and you want to uncomment them, do you see yourself selecting all those lines to uncomment it? Did you know that Ctrl+5 does that promptly for you from any line of that block?

14. Just edited something but don't remember where it was? Ctrl+Q can be used to go to the last place you edited some code.

15. Do you have some code you just copied from somewhere and want to create a module with it? Go to the PyDev package explorer and just paste the code there to create a new module with those contents.

16. Do you have some method call or variable that you want to add to another local variable or field? Use Ctrl+1 (on a line that still doesn't have an assign) and choose assign to local or assign to field.

Besides those tips, print the bindings from: http://pydev.org/manual_adv_keybindings.html and leave it by your side until you don't master them :)

PyDev 3.4.1: improved interactive console, multiprocess debugging

One of the major features added is that now F2 can be used to send the current line to the interactive console (or the current multi-line selection).

It's close to what was previously used as Ctrl+Alt+Enter, but with some differences:

1. It'll fix the indentation of the code when sending things to the console (so, you shouldn't have syntax errors because of the indentation when using Ctrl+Alt+Enter).

2. When a line is sent, the cursor goes to the next line (this may seem minor, but it's a huge time-saver and makes things much smoother).

3. It's a single keypress! (and that was really hard to decide as all keys seem to be taken -- even F2 -- but I think that F2 as it was before wasn't very useful).

All in all, if you're into doing things interactively, things are nicer with F2.

Note: Ctrl+Alt+Enter can still be used to do an execfile, but aside from that, F2 is the preferred way to send contents to the console.

Credit goes to Ed Catmur and James Blackburn for this feature -- I just integrated it :)

Another major feature is that the debugger now works properly when debugging multiple processes (and when a launch is terminated it'll also kill all subprocesses). This means the previous patch the debugger did to Django is no longer needed... Just F11 to relaunch your last launch and debug multiple processes!

Major feature #3: Ctrl+F9 (which opens a dialog for selecting which tests to run) now works properly with py.test too -- even if tests are not under class.

Major feature #4: pxd and pxi Cython files are now properly handled.

And as the last thing, which isn't an actual feature, but something noteworthy: PyDev no longer changes the default encoding (i.e.: sys.setdefaultencoding). Back when PyDev started to do that (in Python 2.4 I believe) there wasn't much choice to see unicode contents properly in the console, so, PyDev changed the default encoding to be the encoding of the console. Fast forward a bit and Python now provides a way to set the sys.stdout/stderr encodings through PYTHONIOENCODING (Python 2.6 onwards), so, PyDev now only sets that variable and no longer changes the default encoding (the main issue there is that when some application was later deployed, it could have a different default encoding and things could break because of UnicodeDecodeErrors).

Note that this is not all, other things were done too... See: http://pydev.org/ for details -- and note that LiClipse: http://brainwy.github.io/liclipse/ is already updated to the latest PyDev too.

Should the Python garbage collector be disabled?

Ok, though question... so, first a little bit of background:

The Python garbage collector is useful for collecting reference cycles, but objects are collected by default when their reference count reaches 0, so, most of the time objects will be collected properly and the collector is only useful when you have a cycle.

Also, there's no guarantee when it'll bump in to do a collection, so, if you're doing UI programming (i.e.: using something as Qt), and you use multiple threads, if you have a cycle, it's possible that the cycle is broken on a collect out of the main thread, which can cause your application to crash if an UI object is collected!

In this case, even if you're careful about collecting things, there's always the case where you have an exception and the object goes to sys.exc_info and becomes alive for much more time than you'd intend, so, if you are using an UI framework, at least making sure that you only collect in the UI thread is a must (see below code which helps doing that).

So, personally, I think that in Python the garbage collector should always be turned off (which can even make your code a lot faster in many situations) and the gc module should be used as a debug tool to find cycles which may occur -- and those should be treated as application errors!

weakref.ref() is one of the most useful things for breaking the cycles and if you need references to methods use a WeakMethodRef: http://code.activestate.com/recipes/81253/

Below is some code to make manual garbage-collection (credit to Erik Janssens) -- while developing the method check() should usually return self.debug_cycles() -- if you want you can use the remaining code to leave as a tool to break cycles in a real application if you want to play safe (although I think disabling it altogether is better if you make sure you don't have cycles) ...

Also, while we're talking about cycles and garbage collection, make sure you never override __del__... Python has an optional callable in weakref.ref() which can be used to do things when an object is collected -- and which doesn't have the problems related to __del__.

class GarbageCollector(QObject):
    '''
    Disable automatic garbage collection and instead collect manually
    every INTERVAL milliseconds.

    This is done to ensure that garbage collection only happens in the GUI
    thread, as otherwise Qt can crash.
    '''

    INTERVAL = 10000

    def __init__(self, parent, debug=False):
        QObject.__init__(self, parent)
        self.debug = debug

        self.timer = QTimer(self)
        self.timer.timeout.connect(self.check)

        self.threshold = gc.get_threshold()
        gc.disable()
        self.timer.start(self.INTERVAL)

    def check(self):
        #return self.debug_cycles() # uncomment to just debug cycles
        l0, l1, l2 = gc.get_count()
        if self.debug:
            print ('gc_check called:', l0, l1, l2)
        if l0 > self.threshold[0]:
            num = gc.collect(0)
            if self.debug:
                print ('collecting gen 0, found:', num, 'unreachable')
            if l1 > self.threshold[1]:
                num = gc.collect(1)
                if self.debug:
                    print ('collecting gen 1, found:', num, 'unreachable')
                if l2 > self.threshold[2]:
                    num = gc.collect(2)
                    if self.debug:
                        print ('collecting gen 2, found:', num, 'unreachable')

    def debug_cycles(self):
        gc.set_debug(gc.DEBUG_SAVEALL)
        gc.collect()
        for obj in gc.garbage:
            print (obj, repr(obj), type(obj))

Configuring Kivy on PyDev

Kivy installation on PyDev is a bit more involved because it requires some environment variables to be set in order to work properly, so, I thought I'd give a step-by-step on where exactly to configure that on PyDev (especially now that LiClipse: http://brainwy.github.io/liclipse/ adds support to the Kivy Language).

Note: The details below use paths based on windows, but it should be similar in other platforms.

1. Download/extract Kivy (for this example D:\bin\Kivy-1.7.2-w32\)

2. Add interpreter in Window > Preferences > Pydev > Interpreters > Python Interpreter (point to: D:\bin\Kivy-1.7.2-w32\Python\python.exe).

Note: For PyDev 3.3.3 onwards, the easier way to go there is doing: Ctrl+3 and writing 'Python interpreter' to open that preferences page (and the same thing can be used to go to a view or even activate some action).

3. Add the Kivy directory to the PYTHONPATH for this interpreter (in the same Python Interpreter page > libraries > add folder > D:\bin\Kivy-1.7.2-w32\kivy)

4. Add 'kivy' to the 'forced builtins' (again in that same page > forced builtins).

5. Add the needed environment variables (in that same page > environment):

GST_REGISTRY = D:\bin\Kivy-1.7.2-w32\gstreamer\registry.bin

GST_PLUGIN_PATH = D:\bin\Kivy-1.7.2-w32\gstreamer\lib\gstreamer-0.10

PATH = D:\bin\Kivy-1.7.2-w32;D:\bin\Kivy-1.7.2-w32\Python;D:\bin\Kivy-1.7.2-w32\gstreamer\bin;D:\bin\Kivy-1.7.2-w32\MinGW\bin;%PATH%

Alternatively, instead of adding those manually to the environment, open a cmd.exe, execute D:\bin\Kivy-1.7.2-w32\kivyenv.bat and then start Eclipse (but then you have to remember to do that manually every time -- or add it to the system environment variables -- note that you have to remember to update it if you move it or upgrade kivy).

After that, it should be possible to go to the pydev package explorer, expand the interpreter node in the tree > system libs > examples, open some example main.py and open it, then, with the editor opened used F9 to run the example (you may have to select which project should be used to get the information on the PYTHONPATH to be used as it's running as an external file).

Changing the locals of a frame (frame.f_locals) and persisting results (with ctypes)

Up until now I didn't know of a proper way to change the locals of a frame (out of the normal execution flow in Python), so, when in the PyDev debugger changing a variable wouldn't always work.

So, for instance, if you have a frame (which you could get from a traceback, sys._getframe().f_back, etc), you could get its locals with frame.f_locals, but changing the frame.f_locals (which gives you a dictionary) wouldn't apply the results back to the frame.

This is mostly due to how CPython works: frame.f_locals actually creates a dictionary using PyFrame_FastToLocals, but changes to the dictionary aren't applied back.

Some years ago I had found a way to make it work (see: http://bugs.python.org/issue1654367) through a CPython function: PyFrame_FastToLocals, but up until recently, I thought it needed a modified version of CPython in order to work, now, recently I discovered ctypes can access a lot from the python api (through ctypes.pythonapi):

So, after changing frame.f_locals, it's possible to use ctypes to call PyFrame_LocalsToFast doing:

import ctypes

ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))

 

A note: the second parameter (which may be 0 or 1) defines whether we want to erase variables removed from the dict (which would require 1) or not.

So,  the PyDev debugger now incorporates this utility so that if you're running in CPython, it will properly change the variable in a scope when you change a variable :)

Note that this isn't compatible with other Python implementations (this is not something the language dictates how it should work -- probably the ideal would be making frame.f_locals writable).