Prototyping a class in Python

I'm not sure what's the name for this technique (for me it resembles what is called prototyping in javascript -- I know it's not the same, but the fact is that you're building your class outside of it, although in Python it's still in its declaration step).

Anyways, the idea is the following: you want to change the class attributes before it becomes final -- there are things you just can't do in Python after a class is already declared (Python uses this technique for creating properties -- in my specific use-case I'm using it to overcome some limitations that Django has in its model inheritance with fields, but I've already used this many times).

The idea is the following: you get the frame which is being used inside the class declaration and change the locals in it so that the final created class will have the things you declared.

I.e.: This code:

import sys

def prototype_class(frame=None):
    if frame is None:
        frame = sys._getframe().f_back

    frame.f_locals['new_attribute'] = 'New attribute'

class MyNewClass(object):
    prototype_class()

print MyNewClass().new_attribute
 

Is the same as:

class MyNewClass(object):
    new_attribute = 'New attribute'
 

-- Just more complicated and more flexible -- in my case, it'll properly help me to choose how to create and customize the fields of a Django model class without having to copy and paste a bunch of code.

On a separate note, blogspot just sucks for code... why can't they simply create an option to add a piece of code? I'm now manually putting my code in html as described in http://stackoverflow.com/a/8697421/110451 -- it'd certainly be trivial for the blogspot devs to put a button which would do that for me right? (colors would be nicer, but this is the easiest things that just works for me and I'd already settle for it if blogger added it -- I know blogger has the quotes, but I want the code at least on a box with a monospaced font).

Python tricks: making sure a function is only called once

Sometimes I want a function to be called only a single time (there are many use-cases, but specifically this time I wanted to change the way the Python garbage collection worked when dealing with Qt and threads).

Anyways, usually I did that setting some flag so that the second time a function is called, it'd check that flag and would skip the function in the second time.


After tinkering a bit about it, I came with a shorter version changing the function code which I thought is pretty nice:

 def func():  
   print 'Calling func only this time'  
   func.func_code = (lambda:None).func_code  

Now, calling it a second time will actually execute the lamda:None code.

Yes, I know you'll say it's cryptic, so, I'm including a decorator version below which does the same checking with a variable:

 @call_only_once  
 def func():  
   print 'Calling func only this time'  
 def call_only_once(func):  
   def new_func(*args, **kwargs):  
     if not new_func._called:  
       try:  
         return func(*args, **kwargs)  
       finally:  
         new_func._called = True  
   new_func._called = False  
   return new_func  

Plugins in Eclipse to accompany PyDev

Below are some plugins which I believe may be very helpful for people installing Eclipse/PyDev (those are the plugins I usually install on any new Eclipse install).


1. StartExplorer: 

Details at https://github.com/basti1302/startexplorer

Mostly, I use it to open the current file in explorer or copy the current file path to the clipboard (but it has other niceties too).


2. Extended VS Presentation plugin for Eclipse

Details at: http://andrei.gmxhome.de/skins/index.html

I use it for skinning Eclipse (i.e.: providing a better look and feel), but note it's only for Eclipse 3.x (Eclipse 4.x has a skinning engine builtin).


3. Eclipse Color Themes

Details at: http://eclipsecolorthemes.org

Note that if you're using the Aptana Studio editors, you don't really need this plugin as Aptana Studio itself has a theming feature which supersedes this, but if you're only on Eclipse/PyDev and other editors, this may be a nice addition so that you can apply the same colors for several editor plugins at once.


4. Practically Macro

Details at: https://sourceforge.net/projects/practicalmacro

Yes, I know, Eclipse does not have macro record/playback by default and it's a major shortcoming. Well, this plugin does cover most of the use-cases I have for doing macro record/playback (although it does have some weirdness sometimes -- in which case I usually just start notepad++ editor just for this feature).


5. EGit

Details at: http://wiki.eclipse.org/EGit/User_Guide

Note that Aptana Studio has its own Git integration (so, when I'm in Aptana Studio I don't usually install EGit). Although it can do many things with Eclipse, I use the Eclipse integration (either from EGit or Aptana Studio) mostly to have a quick way to know which files I've changed and sometimes seeing the history for a given file. For most of my work I like to work on the shell, usually using a tool I wrote in Python: https://github.com/fabioz/mu-repo which helps managing many git repositories at once from a shell (and showing differences I can edit via WinMerge) -- and sometimes I even fire http://code.google.com/p/gitextensions for exploring some git repository.

I was thinking a bit about this one since in cvs and svn I did everything inside Eclipse, but when it comes to git this trend didn't continue. I think it's mostly because the major thing for me is that the synchronize view wasn't available to me on cvs/svn and the EGit synchronize view is not as streamlined/fast as the other integration -- as I have the habit of reviewing all the code I'm about to update, and as EGit made that process a bit slower (and git is pretty easy to extend with scripting on top of it), I just went a different way this time (but who knows, if EGit provides a more streamlined/faster synchronize view in the future for my modus-operandi I may switch back to only Eclipse here -- but it'd have to beat the time in which I do things in mu-repo which fetches/diffs lots of repositories in parallel in a pretty fast way, so, I'm not sure how feasible that is).

Debugging SQL in Django (with PyDev)

I'm working on a Django project recently and one of the things that came up was debugging SQL statements a bit to see what was actually going on.

After researching a bit, I found: http://pypi.python.org/pypi/django-debug-toolbar, which when installed wraps around the calls that Django does to the DB and prints the actual SQL statements used.

I didn't really install it as a debug toolbar as it recommends, but I use it in the interactive console shell to see how things happen when I'm experimenting.

To do this, after installing it and adding 'debug_toolbar' to your INSTALLED_APPS in Django, one can start the interactive console shell with Django support (if you're in a Python editor just do: Ctrl+2 dj shell) and add write the import:

from debug_toolbar.management.commands import debugsqlshell


Now, when you do some operation that does an SQL operation, it'll be printed to the console, while having all the PyDev niceties on the console (such as code-completion). Pretty nice.

PyDev & Eclipse 4.2 (i.e.: stick with Eclipse 3.8)

Ok, I've been giving a whirl on supporting Eclipse 4.2 on PyDev and everything seems to work as expected, but unfortunately, I still can't recommend the Eclipse 4.x series over the 3.x series because performance-wise Eclipse 4 is still not on par with Eclipse 3 (and for me the only gain was that floating docks are now better -- as for skinning, I don't really think it's much better for final users unless someone else already got to the point of doing a skin exactly as you'd want -- so I think that for now, Aptana Studio 3 skinning (themeing) is a better solution -- and targeted at Eclipse 3.x).

I have hopes that those issues will be addressed in a newer release of Eclipse 4, but the truth is that right now Eclipse 4 feels sluggish when compared to Eclipse 3 (even changing configurations on the Eclipse appearance preferences for having some performance gains).

So, if by any chance you've just gotten into PyDev and it doesn't seem as speedy as you'd like, please make sure you check it with Eclipse 3.8 (in the downloads page: http://download.eclipse.org/eclipse/downloads/ there's a link to Eclipse 3.x downloads available: http://download.eclipse.org/eclipse/downloads/eclipse3x.html).

Also, make sure that you have the latest version of java as it may make a difference.

Or, if you've gotten the Aptana Studio 3 download directly, you're already good to go -- as it's already based on Eclipse 3.x and has PyDev preinstalled (see http://pydev.org/download.html for links).

Git (multiple repos)

Improving my Git workflow has been one of my current targets lately... Especially when working with multiple projects.

So, although I'm working all the time inside Eclipse, I still like to use the command line for many things (so, even with the Aptana Studio 3 Git integration and EGit having improved, I still use the command line a lot on common operations, resorting to other options mostly when investigating the history of a repository or some other thing which may be suboptimal in the command line).

To improve that situation, I ended up creating a tool... Yes, I know there are probably many other command line tools that try to accomplish that, but I did a good amount of research and didn't find anything that worked as I wanted -- and not nearly as fun (but I appreciate suggestions in this realm).

The tool I did is hosted at GitHub: https://github.com/fabioz/mu-repo, and it's freely available (GPL 3).

It's a Python program which interfaces with the git binary to execute whatever is needed. Also, for diffing operations it currently relies on WinMerge (meaning this part of the tool is Windows only for now, but if there's interest, it should be straightforward having it support other diff tools such as KDiff3).

My basic workflow is now:

cd /workspace (which has many .git subrepos)

Note: all the "mu" operations below will actually be applied to all the git subdirs (must be previously registered), so, a single command line is enough for dealing with all repositories at once -- yes, I know about git submodules, but I don't really like how it works -- and it's one of the main reasons this tool exists :)

To get changes:

mu up (fetches origin for the current branch updating the origin/current_branch reference -- I found it weird that trying to do "git fetch origin current_branch" ended up updating FETCH_HEAD instead of origin/current_branch -- also, this and most other operations work only in the current branch, leaving other branches or references untouched).

mu dd origin/current_branch (preview incoming changes: compare current branch HEAD to origin/current_branch).

mu rebase origin/current_branch (rebase incoming changes -- could also be: mu merge origin/current_branch).

To submit changes:

mu st (check status of changed files -- indexed or not)
mu dd (diff working copy with winmerge while editing changes and keeping the working dir updated even if no symlink support is available)
mu acp "message" (add all working copy changes, commit, push)

To start working on a feature:

mu co -b branch_name (create branch for all repos)
mu ac "message"  (add / commit changes on that branch)
mu co main_branch (go back to main branch)
mu rebase branch_name (get the changes from the branch into the main branch)
mu branch -D branch_name (remove that local branch)



Note that for my workflow I work mostly without taking "advantage" of the git index -- usually I review things with "mu dd" and on "mu ac" I add all the changes to the index and commit in a single operation (before that I was almost always doing "git add -A" and "git commit -m "message"" anyways).

I also still configure many things as I posted previously: http://pydev.blogspot.com.br/2011/02/git-on-command-line.html, but a bunch of things are now better integrated with this tool :)

Also, the tool works for a directory that has a '.git' repository, so, operations such as the diff with WinMerge can be used even when working with a single git repository.

p.s.: The tool is still in beta, so, although I'm using it all the time, it's possible that some things are not working as they should -- but on the good side, I still haven't been able to corrupt any of my git repos :)

PyDev 2.6.0 released

PyDev 2.6.0 is just out of the door.

Main in this release were changes in the interactive console -- and mostly by external contributors, so, it seems this feature has really picked up some traction :)

Hussain Bohra made it possible to have an interactive console bound to a debugger frame and James Blackburn did a bunch of fixes in it.

Aside from that, the default PyUnit runner has new options to include/exclude files from the test loading process and Jython has an annoying fix done (print can be used as a dotted notation) along with many other fixes (see: http://pydev.org for more details).

Also, the PyDev official issue tracker is now the same tracker used in Aptana Studio 3: https://jira.appcelerator.org/browse/APSTUD/ (there's a component related to PyDev there).

The issues currently on Sourceforge will be migrated to that tracker, but there's no way to make the current Sourceforge trackers read-only, so, I'm just redirecting people to the new tracker and asking not to post on the old one -- hope that works :)

It's interesting to note that now the official Eclipse release is focused on Eclipse 4.x (and not on 3.x)... the integration has already been tested, but not as much as the 3.x series, so, please report any issues found in the way.

PyDev 2.5.0 released

Major things in the PyDev 2.5.0 release include:

• Proper support for Django 1.4 (as it's layout changed, the project creation wizard wasn't working properly).

• Better patching of Django 1.4 for the debugger and auto-reload (details at the end of the page: http://pydev.org/manual_adv_django.html). Note: there is an issue when using it on Linux, which is already fixed in the nightly build, so, if you are on Linux and want to experiment with it, please update to the current nightly build.

• The interactive console can now be attached to the variables/expressions view (the preference at: window > preferences > PyDev > Interactive Console and check 'Connect console to Variables Debug View?' must be enabled). Also, there's drag and drop support for it now.

Aside from those, many bugs were fixed and minor improvements were done. More details can be seen at: http://pydev.org/

PyCon retrospect

Ok, so, this was the first time I've been on PyCon and I must say it was pretty nice.

Initially, it started out kind of lousy for me: United seems to have 'misplaced' my bag and I only got it back in time to check it back to Brazil, so, that sucked. On the other hand, to balance things, I won a Kindle DX in raffle from http://www.truecar.com :)

Mainly, it was awesome talking to so many great names of the Python world... Meeting people face to face does make a great difference (some things are simply not as well expressed in an e-mail).

On the PyDev side, it was pretty nice seeing many people using it while hacking away during the conference and even having some people asking 'how can I help to make PyDev better?'.

I guess I wasn't properly prepared and mostly, my answer was: read the devs guide to grab the code/compile, find an itch to scratch and mail me about it so that I can help you find yourself there... Some things can be done in Java, some in Jython, depends a lot on what you want to do.

Aside from that, documentation-wise, I guess the major issue would be having some section explaining how to configure each major library on how to work with PyDev (most work out of the box, but some need minor adjustments such as adding a token to the forced builtins and some may even require more than that), so, if someone is willing to help working on the PyDev documentation in that topic, that would be a great help (and I guess that if someone doesn't have an itch to scratch and still wants to contribute in code, there are things I can point to that need to be done and are relatively simple to get started).

On the scientific computing side, I liked Numba (https://github.com/ContinuumIO/numba), which seems to be a pretty nice idea... it's still in its early stages, but I think it could be a nice substitute for Psyco (but with knowledge about numpy types and maybe a bit more strict -- if I did understand it properly)... I'd much rather use something as Numba for speedups than Cython (as the Cython code is only 'close' to Python, but doesn't really run on a Python interpreter) and Pypy is just not an option right now...

Although Pypy seemed strong in the conference, I must say that it'll probably be a long, long time until I'll be able to use it in production... I really use lots of code in C/C++ bindings, so, this is a killer CPython feature which I'm not sure Pypy will ever be able to provide (and the 'current' workaround of porting a library to RPython as is being done in Pypy/Numpy doesn't seem to be really feasible).

One thing I thought was a bit strange was the lack of projects with Python in mobile platforms, as I was guessing it should be straightforward to compile Python to Android/iPhone (so, I'm guessing the issue is probably the lack of proper multi-platform bindings on that area). In contrast, Python seems very strong on the server-side and scientific computing fields.

And I think Python 3 does deserve a special note here: I think Guido left it pretty clear (as much of his keynote talked about the subject) that Python 2 is really deprecated, so, although it's currently the production version that's most widely deployed/used, people have to really start getting used to the idea that a port to Python 3 is the way to go -- although in the real world, I still do expect this to take quite a few years (probably more than was initially expected by the Python-core folks). The fact that Guido made this topic such a big portion of his keynote, does make me feel like many of the Python lovers were against this move and the actual response from the community on this respect has been mixed, but regardless of that, the direction that Python-core is going is 100% on Python 3 (maybe doing something here or there to help in easing the port or having a shared Python 2-3 codebase).

And, there are still have some talks I'll see later at: http://pyvideo.org (because many interesting talks took place at the same time).

PyCon

Tomorrow I'll be traveling to PyCon and I'll be staying during the 3 days of the main conference (Friday to Sunday).

So, anyone who wants to talk about PyDev, scientific programming, how to best structure a Python development environment -- or just wants to say hi, I'll be there this time :)

Just to note, currently the major projects I'm working on are:

PyDev (http://pydev.org), which I guess is pretty well known (especially if you're reading this blog), and I've been developing it for 9 years already :)

and Kraken (https://www.esss.com.br/kraken/), which is a software to do post-processing of reservoir simulations (this may be less known, but actually, I started working on PyDev to scratch my own itches while doing scientific programming in Python, and so far, Kraken is the most advanced/complex software I've worked on this area and I've been involved with its development since its beginning, around 5 years ago).

See you in Santa Clara :)

PyDev forums -> StackOverflow

The PyDev forums at SourceForge are now officially deprecated :)

So, anyone having a doubt regarding PyDev should now ask at StackOverflow and add a 'PyDev' tag.

I think this will be a real improvement over the current status quo... Some reasons I see for that are:

1. Many PyDev users follow StackOverflow and do answer things there, whereas in the PyDev forum, many questions were asked, but I was almost the only one answering... (I think the real plus here are the 'gaming' features that StackOverflow has, so, more people are inclined to participate actively).

2. As people started asking there anyways, I really had to follow StackOverflow closely too, so, deprecating the PyDev forums means I'll be able to follow a single place again :)

3. Interacting with StackOverflow as a whole seem a nice improvement over the SourceForge forum (it's edition is nicer, accepts pictures, etc.)

And now on to something a bit unrelated... the PyDev homepage (http://pydev.org) is now being generated from a wiki (it's still a read-only wiki -- but hopefully that'll change soon -- but at least, I feel it'll be easier for me to edit things there and later have the homepage updated from it). So, if someone finds something strange in the homepage, please let me know :)

PyDev 2.4.0 released

This release was mostly focused on performance and memory optimizations.

On the performance front, the major focus was on start up (i.e.: start up Eclipse, open an editor, request a code-completion and show the globals token browser (Ctrl+Shift+T)) -- which should've become pretty fast (tested only with the Eclipse runtime and PyDev -- as things in other plugins can't really be controlled -- the subversive plugin for subversion seems to be especially slow to startup).

Memory-wise, things have been improved too, with the AST taking up less memory and doing a 'pseudo-intern' for some rather large caches (it's a pseudo-intern because the String.intern() function is not used: a HashMap is done and strings are reused inside it for some processes -- and later that HashMap is thrown away), and the Jython plugin was fine tuned to make less plugins visible to save on memory (and startup time).

Just to note: the real memory used can be seen going to window > preferences > general > show heap status (the real size of the java process in the OS will probably be bigger as java will usually grow to the size specified by -Xmx, regardless of how memory it's really using at a given time). Personally, on large projects I allocate 300 Mb for the process, but this is mostly because the subversion plugin seems to be rather resource hungry -- migrating to git on some of those projects seems to be making things better :)

Aside from that, this time I spent some time migrating the PyDev homepage to a wiki ( https://wiki.appcelerator.org/display/tis/Python+Development ) -- right now it's not available for external edition, but that should happen soon (hopefully), and the idea is that the PyDev homepage will be generated mostly from that wiki.

And as usual, a bunch of bugs were fixed :P

Code-completion strategies in PyDev

I believe one of the strong points in PyDev is its code-completion, so, I thought a bit about giving some details on it :)

The main preference page for code completion is: Window > Preferences > PyDev > Editor > Code Completion (my preferred configuration is setting the 'Request completions on all letter chars and '_'', so that completions appear automatically when typing, otherwise Ctrl+Space would need to be used to request the completions -- I was actually thinking about making that the default and decided against it to conform to other editors in Eclipse).

* Word completion (also called Hippie Completion):

This is probably the simplest one and is provided by Eclipse itself (through Alt+/). It provides a simple word-based completion which uses all the currently opened editors in Eclipse.

I've actually provided a patch for Eclipse to improve the speed of this completion (https://bugs.eclipse.org/bugs/show_bug.cgi?id=270385), which was added in Eclipse 3.6.

* Templates completion:

These are user-defined templates that may be configured at PyDev > Editor > Templates (most of the base for this completion is provided by Eclipse... PyDev uses a subclass: PyTemplateCompletionProcessor and some of the available variables may be defined in Jython code -- see: pytemplate_defaults.py for details).

* Common tokens completion:

When you start typing in PyDev, some common tokens (i.e.: keywords, self, etc) start appearing directly. Those can be configured in PyDev > Editor > Code Completion (ctx insensitive and common tokens).

It's implementation is pretty simple (may be seen at: KeywordsSimpleAssist)

* Context insensitive completion:

This completion goes through all the tokens available for a given project (which may need to consider project dependencies and which interpreter is being used) and shows those tokens as a completion (i.e.: top-level tokens such as classes or methods and the modules themselves).

If one of those is selected, the token will be completed and an import will be added for it too (if the preference in PyDev > Editor > Auto Imports > "Do auto import?" is marked as true -- in that same preferences page, the number of chars that need to be available in a word so that these completions start appearing may be specified).

Note that if the option was set not to do the auto-import, one could just add the token, let it be marked as an unrecognized variable by PyDev and later do an Organize Imports (Ctrl+O), or a Quick Fix in that line (Ctrl+1), to add the import.

The major issue in this completion isn't actually the completion per-se (implemented in ImportsCompletionParticipant and CtxParticipant), but the structure which needs to be kept to have it as a fast and efficient completion.

Mainly, PyDev has a concept called 'AdditionalInfo' (this was done when PyDev Extensions was separated from the PyDev Open Source, so, the name is a bit strange now, but the general idea is that it was additional information related to a given project or interpreter), which keeps the following information:

- Two TreeMaps (AbstractAdditionalTokensInfo.topLevelInitialsToInfo and AbstractAdditionalTokensInfo.innerInitialsToInfo) which map token names to information of the places where the token may be found (i.e.: module and structure inside that module). Those are all kept in memory and are pretty fast to access (AbstractAdditionalTokensInfo.getTokensStartingWith is what's interesting for a code-completion and AbstractAdditionalTokensInfo.getTokensEqualTo is interesting when doing a quick fix or organize imports). This structure is also used in the global tokens browser (Ctrl+Shift+T).

- Note that it also has a structure (AbstractAdditionalDependencyInfo.completeIndex) which maps a module to all the available tokens in it. This structure is kept in memory only as a SoftHashMap (so, it's only kept in memory while there's enough space for it) and persisted to the disk. It's also only lazily created on operations that need it (currently only a project-wide rename refactoring or a find references (Ctrl+Shift+G) would use it as it's basically a structure which is a bit faster for doing exact match searches than actually doing a search in Eclipse -- especially if the SoftHashMap is still in memory, so, if many find references are done in succession, if there's enough memory, from the 2nd attempt onwards, things should be fast).

On a project build, the tokens of the completeIndex are simply all removed (to be recalculated when some action that needs it is called). As for the maps, those are always kept up to date when a file is changed. The strategy for having it build fast is that the in-memory cache is directly updated (which is reasonably fast) and instead of saving the whole map it just saves the delta information and when restoring the info, those deltas are applied to have it in the last state (and from time to time it does dump the whole structure and removes the deltas). Also, it runs in a separate thread (not actually in the thread that's doing the build, and a singleton: RunnableAsJobsPoolThread, makes sure than only some of those, depending on the number of processors in your machine, are running at the same time, so, if you change 200 files at once, your computer won't come to a halt).

* Context sensitive completion:

This is by far the most complex completion available as it analyzes the context where you're requesting a completion and provides tokens based on it. Basically, PyDev has an internal type-inference engine to do that (which is also used by actions such as find definition or TDD actions such as create method).

Internally it uses an LRU structure which maps module names to the module AST (Abstract Syntax Tree) and in a pretty recursive algorithm finds out about the available tokens needed for a given context and provides completions based on that (thankfully it has a huge amount of unit tests holding it all together). That process starts in PyCodeCompletion.getCodeCompletionProposals(ITextViewer, CompletionRequest) and the type inference engine main classes are: ASTManager and ProjectModulesManager.

On some occasions some modules may be pretty hard to analyze, in which case PyDev resorts to launching a shell and querying it for the needed tokens (those are pre-specified as in window > preferences > PyDev > Interpreter > Forced Builtins, and the communication happens in the java side through the AbstractShell class) -- it's also probably one of the main reasons of problems when configuring PyDev, as it's common to have a firewall blocking that communication (in which case PyDev wouldn't even be able to get common builtins such as len, object, etc).

On the good side, this also makes it possible for PyDev to analyze .pyd modules (although if you're developing such a module as a part of your project, you have to remember to call Ctrl+2, kill so that PyDev will kill those shells before you actually build it, otherwise that module will be locked and you won't be able to link it -- and tokens wouldn't be updated).