PyDev 4.0 released. Yay!

PyDev just reached version 4.0 --- after 11+ years that's not so much, is it? ;)

The major improvement in this release is that PyDev can now unpack compound types when doing code-completion (i.e.: list(str), dict(int:str), etc).

The screenshot below shows a completion unpacking types by analyzing the current scope:

The second example shows a completion unpacking types by analyzing a docstring (see: http://pydev.org/manual_adv_type_hints.html for more details on the formats supported).

This release also enables Unicode literals and Byte literals to have different colors, so, it's easy to spot what you're working with (and it should respect the semantics of Python 2 from __future__ import unicode_literals and Python 3).

Another nice feature for those that work more in the interactive side of Python is that PyDev now allows users to define a template to send a custom command to the interactive console.

PyDev already has an F2 binding which sends a line to the console, but let's say that you do something else a lot in the console (such as plot(variable)). With this feature you can bind something as Ctrl+F2 to send the selected text as plot(${text}) to the console (the screenshot below shows how that configuration would be achieved).

Also, the PyDev update site is now hosted on http://bintray.com -- note that the update site urls didn't change, so, this change should (hopefully) be imperceptible to end-users...

There are also other nice things (see http://pydev.org for more details).

To finish, a special thanks to all the PyDev supporters (it definitely wouldn't be possible without your help -- for those that want to help supporting it, please access the related links at http://pydev.org).

Navigating through your code when in PyDev

One of the main advantages of using an IDE is being able to navigate through your code in a really fast way, so, I'll explain below how to find what you want when coding in PyDev.

1. Any file in your workspace

The simplest way is searching for any resource for any project you have available (this is actually provided by Eclipse itself). So, if you know the file name (or at least part of it), use Ctrl+Shift+R and filter using the Open Resource dialog:

2. Any Python Token

Another way of getting to what you want is using the PyDev Globals Browser (using Ctrl+Shift+T).

This is actually what I use most as a single place can be used to filter for package names, classes, methods and attributes inside your own projects or any other package in Python itself. Also, it allows for advanced filtering, so, you can search for tokens only inside a package (i.e.: dj.tz filters for any 'tz' token inside django in the example below).

3. Quick Outline (current editor)

Ctrl+O will show a Quick Outline which shows the structure of your current file. And pressing Ctrl+O one more time in this dialog will also show the structure of superclasses in the hierarchy (you can see that in the example below __setitem__ appears twice, once for the method in this class and another one for the superclass).

4. Selection History

Go back and forward in your selection: Alt+Left goes to the place you were before and Alt+Right to the place you just went from... this allows you to easily go navigate through your recent places.

5. Open files

To filter through the open files you can use Ctrl+E: a dropdown will appear and from there you can filter through its name and you can close existing editors using Del from that dropdown too.

6. Go to previous next token (class or method)

Ctrl+Shift+Up and Ctrl+Shift+Down allows you to quickly navigate from your current position to the previous or next method (selecting the full method/class name).

7. Navigate through occurrences and errors in the file

Ctrl+Dot allows navigating through occurrences and errors found in the file, so, in the case below we'll navigate through the occurrences of 'func'.

8. Go to some view/menu/action/preference

This is an Eclipse standard mechanism: using Ctrl+3 allows you to navigate to any part of the IDE.

9. References

Ctrl+Shift+G will make a search showing all the references to the token under the cursor (and the search view where results are shown can be navigated with Ctrl+Dot).

10. Go to definition

Just press F3 or Ctrl+Click some available token and go directly to the selected place.

11. Hierarchy View

Using F4 shows a hierarchy view where you can see the structure of your classes.

12. Show In

Alt+Shift+W allows you to see the current file in a given place (such as the PyDev Package Explorer or the System Explorer from your OS) or your current class/method in the Outline View.

Others

The ones below are available in standard Eclipse and you should also definitely know about it :)

Ctrl+L allows you to navigate to a given line in your current editor.
Ctrl+Q goes to the place where the last edition was made.
Ctrl+F6 navigates through the opened editors. In LiClipse Ctrl+Tab is also bound to it by default-- and I suggest you also add this binding if you aren't using LiClipse :)
Ctrl+F7 navigates through opened views (i.e.: Package Explorer, Outline, etc.)
Ctrl+F8 navigates through opened perspectives (i.e.: PyDev perspective, Debug perspective, etc).
Ctrl+F10 opens the menu for the current view (so you can select filters in the Package Explorer, etc.)
F12 focuses the editor (so, you can go from any view to the editor)
Ctrl+H Opens the search dialog so you can do text searches
Ctrl+Shift+L twice goes to the keybindings preferences

Now you can enjoy going really fast to any place you wish inside PyDev!

Appcelerator, Aptana & Pydev

Well, as publicized already, Aptana was acquired by Appcelerator (http://www.appcelerator.com/appcelerator-acquires-aptana/).

So, I thought I'd write a bit about it...

First thing to note is that (as the rest of the team) I've joined Appcelerator, and the plan is to keep Pydev going forward (as it has been going since I had joined Aptana). However, unlike the rest of the team, I'm still based on Brazil :)

I think that the only difference in focus is that now, aside from the support for Django and Google App Engine, Titanium (which is Appcelerator's platform for developing desktop and mobile applications) will also have some special attention, although I still don't have more details on what exactly this means...

All in all, I think this was a good thing, as one of the things that Aptana was looking for was something to support (as the plans were always about giving Aptana Studio for free and making money on something else), and Appcelerator will have a nice improvement in terms of a better coding environment :)

Pydev and virtualenv

A doubt that arises sometimes is how to use Pydev with virtualenv, so, here we go...

For those that don't know about it, virtualenv is a Python tool that allows one to configure a new Python interpreter that's 'sandboxed' based on an existing interpreter.

Sample use of virtualenv:

Say you have an interpreter in d:\bin\python26 (with easy_install in place).

You could do d:\bin\python26\scripts\easy_install.exe virtualenv to install virtualenv and later do:

d:\bin\Python26\Scripts\virtualenv.exe d:\bin\Python26django, which would create a new Python install at d:\bin\Python26django.

Later, you could add new libraries just at that Python install without disturbing the base install.

E.g.:
d:\bin\Python26django\Scripts\easy_install-2.6.exe django would install django at that interpreter without changing the default one.

Now, as far as Pydev is concerned, there's nothing special in configuring a regular interpreter or an interpreter available through virtualenv. It's just a matter of following the default steps (available at http://pydev.org/manual_101_interpreter.html) but pointing to the interpreter created by virtualenv (in this case d:\bin\Python26django\scripts\python.exe).

Note: For linux users, the instructions should be similar, but the created folder by virtualenv doesn't put the interpreter at a 'scripts' folder, but at a 'bin' folder.

Update: Note that when selecting a virtualenv interpreter, the folders from the base install won't come checked by default when adding the interpreter, so, those have to be manually selected.

Django on Pydev (take #2)

It's nice that the previous post on Django on Pydev made so many people happy :)

There were many comments and I decided for writing a separate post to answer those and explain a bit better what should be the final state of the integration.

1. Django templates

There will be a special editor for that, but this will not be in the next release. This editor will be based upon the new Aptana Studio 3 (if you don't know about it, there's a preview explaining it at http://vimeo.com/channels/radrails#9895797). Initially it may be simpler (at least syntax highlighting and outline) and it should shape to have code-completion and other features in the future.

2. Custom Django actions

A new nightly has just been pushed with support for any custom action to manage.py. There's still room for improvement, like automatically going through the manage.py help and executing a command or asking help on that, but this won't be available for the upcoming release.

3. Code Analysis of Django

Most things should work as it is now, but I'm waiting for bug-reports saying what exactly Pydev fails to acknowledge on Django.

And that's it for the current plans... if there are more suggestions, they're very welcome!

Templates on Pydev (with Jython scripting)

One really nice feature that has just been added to Pydev (available in the nightly build) is the possibility of extending the variables that are available for templates through Jython scripting.

It opens lots of possibilities, as when it's resolved it has access to the python code being edited (and all the available modules and Pydev APIs).

Some default templates were added already taking advantage of that (see: pytemplate_defaults.py as a reference -- the docstring explains how to create your own templates).

The image below shows the variables created in the defaults (already using Jython scripting -- note that when multiple superclasses are available, the ${superclass} variable enables the user to choose which one to use):





This enabled the print template to work as expected (giving only 'print ' in python 2.x and 'print()' in python 3.x, as it can 'know' which grammar you are using).

To finish, 2 other templates were creating taking advantage of the context:
super : super(MyClass, self).CurrentMethod()
super_raw : MySuperClass.CurrentMethod(self)

If anyone has other 'must have' template suggestions now that this is available, those are very welcome.

On a side note, the current nightly build is pretty stable (and should be the new released version unless there's some critical error lying there found in the next 2-3 days), so, it should be safe to get it to experiment with those.

Refactoring / AST rewriter in Pydev

Right now, there's a bit of quietness in the Pydev realm... I'm working on the current refactoring engine -- which is right now based on the PEPTIC refactoring engine, BRM and the previous Pydev Extension code.

The idea is to stop using BRM altogether and use the others to make up an improved refactoring engine (as that's a big task, I'm avoiding pushing it until it's not stable).

I decided that the first step to do that would be getting the latest PEPTIC code and making it work with the current tests (I had already done some changes to the current Pydev version, so, it diverged a bit).

So, while starting to fix those tests, I realized that its AST rewriter did not properly support the Python 3.0 grammar, so, this is where I am right now, making a decent rewriter. There are 2 AST-based rewriters right now, one that needs the parsing tokens together with the AST and one that only needs the AST -- needles to say none works the way I think they should, so, I decided to go on a different approach, creating a new rewriter.

This rewriter will work with the tokens if they are available (which will make it able to maintain the comments in the correct places) and will generate tokens if they're not available.

I've decided on the following approach: instead of just parsing and writing things to a string in a single step (as the current rewriters do), it'll traverse the structure and fill an auxiliary document structure with the meta information found (i.e.: token requests at a given position and the actual tokens found -- if available -- plus information on the AST, such as statement start and end, nodes, etc), and in a 2nd step, it will generate the actual python code by traversing this document structure. A problem in the previous approaches is that sometimes it found a token and then saw that a comment should be added before it, or a token would not be in the expected AST (and it was really difficult to go back to fix it) -- so, I hope this won't be a problem anymore :)

Unfortunately, as I said, this is a big task, so, I don't think I'll be able to make a release this month -- but looking on the other way, the current release seems pretty stable :)

Git (and Folder compare with Git with WinMerge)

[Update] there's a new post with instructions for configuring git at: http://pydev.blogspot.com/2011/02/git-on-command-line.html

In case it was missed, Pydev is now on Git :)

I'm finding it the hard way that unfortunately its eclipse integration is still not ready for prime time -- it's missing some basic things such as a synchronize view, and I've had some bugs for it to mark the changed files in the package explorer (so, I'm waiting until it matures a bit to give it another try).

In the meanwhile, I'm using the command line (with the msysgit bash).

The thing I missed most was the synchronize view. A part of my modus operandi is making a review of all the changes before doing a commit -- and editing some things there, so, based on this link, I've created a script -- which I called gitdd -- that will create a directory structure with the HEAD and links to the changed files so that one can change those files in a diff viewer (in this case, WinMerge).

The script is below if anyone wants to use it (note that it must be run in the msysgit bash -- although it should probably work on Linux too, with another diff viewer)


#!/bin/sh -e

# usage: gitdd <path>
# Compares the current differences in winmerge with links to original files.
# Note that it must be executed at the root of the git repository.

SUBDIRECTORY_OK=1

O=".git-winmerge-tmp-$$"
V=HEAD
list="$O/list"
list_exist="$O/list_exist"
trap "rm -rf $O" 0
mkdir $O
mkdir $O/b

git diff $V --name-only -z $1 > $list

/d/bin/Python261/python.exe /c/Program\ Files/Git/bin/gitddpy.py $list $O/b/

cat $list | xargs -0 git archive --prefix=a/ $V | /d/bin/bin-1.0/tar.exe xf - -C $O

/d/bin/WinMerge/WinMergeU.exe //r //u //wr //dl WorkingCopy $O/b $O/a


[Update]
I wanted to create links to the files (so that I could edit them on WinMerge and have the original changed), so, I ended up creating a gitddpy.py to create the link (not sure why git prompt was crashing when using the shell for that -- so, this was the solution that worked for me).

The gitddpy.py contents are:


import sys
print sys.argv

f = file(sys.argv[1])
contents = f.read()
import subprocess
import os.path
for line in contents.split('\0'):
    if os.path.exists(line):
        subprocess.call(['cp', line, '--parents', '-l',
        '--target-directory='+sys.argv[2]], shell=True)
f.close()



[Update]
Configuring Git with the following:

git config format.pretty "%h %ct %ad %Cgreen%aN%Creset %s"
git config log.date short

The commands I use most (using the msysgit Bash) are:

git status
git commit -a -m "Message"
git push origin master
git checkout <path>
git log -n 6 --format="%ct %Cgreen%aN%Creset %s"
git commit --amend (change last commit message)

And some of the commands I had to discover how to use in the msysgit bash are:

Alt+Space+E+K: mark contents for copy
Insert: paste contents
Alt+Back: erase until whitespace

The last thing worth mentioning is that the Pydev releases don't have such a nice unique id as the one that was available with subversion anymore, and it now contains the commit time (%ct in the log format). It's a bit annoying that it's such a big number, but I believe that the other choices were worse (the hash would seem totally random for someone seeing it, and basing it on the number of commits wouldn't work with history rewriting).

To sum things up: yes, I know that not being able to do everything in Eclipse sucks a bit, but hopefully the Git tools for Eclipse will catch up soon -- I'm hoping that having the history offline and not having to worry about renames, as well as the whole idea of decentralized versioning systems pays for this nuisance for now (and I'm certain that when the Eclipse plugin for Git catches up it'll be much better).

New Pydev release soon (call for testers)

A new Pydev release should be available pretty soon... if no critical bugs are found, the current nightly build -- version 1.4.5.2712 -- will be the released version.

So, whoever is interested in the next release, please get the latest nightly build -- http://pydev.blogspot.com/2009/02/pydev-nightly-builds.html -- and check it to see if everything is working fine (so that I can provide fixes for any critical bug found before it's actually officially released).

Note that this is an attempt to have released versions more stable -- because even with a good test suite and some days of manual testing before an actual release, now and then there are still critical bugs slipping (there are way too many different scenarios and use-cases in which Pydev is used)

Sure, it's nothing like the 1-2 months of testing Eclipse goes through before a release... Actually, from submitting something up to approval in Eclipse, it usually means 6-10 months for it to be in a final release... for instance: I've just submitted a rather minor patch to make the word completion (Emacs-Style) faster -- but it'll only be patched for Eclipse 3.6... meaning that with luck it'll be available 10 months from now (because it's in API freeze now and in 2 days will stay at feature freeze and will get its 1-2 months of testing) -- but on the bright side, I can patch my own version :)

Pydev 1.4.4

Just now I realized (thanks to Radim) that I forgot to post about 1.4.4 (which was released some hours after 1.4.3).

Pydev 1.4.4 was a one bugfix release: it fixes a critical bug when configuring the interpreter (if no environment variables were specified, it was not possible to configure an interpreter).

So, if you got 1.4.3, please upgrade to 1.4.4! (1.4.3 was removed from the update site because of that bug).

Pydev Nightly builds

Yeap, it's available for those that really want to keep up with the most up-to-date code...

The nightly build is available in the format of an update site.

The update site links to use are:

http://nightly.aptana.com/pydev/site.xml
http://nightly.aptana.com/pydev-pro/site.xml

Instructions on how to use it are available at:

http://nightly.aptana.com/pydev/index.html
http://nightly.aptana.com/pydev-pro/index.html

Until now, Pydev didn't have an easy way for people to get updates until a proper release was all wrapped up, so, this should be pretty handy for those of you expecting a bugfix -- or just wanting to experiment with the latest thing you're waiting for. Also, usually before committing I do run the Pydev test-suite (which is pretty big), so, it should be regularly pretty stable.

Another thing: I've been having some complains on people not being able to access fabioz.com, even with the server being up (it appears it's related to some specific providers), so, that update site can be used as a second choice if you're not currently able to access fabioz.com to get Pydev Extensions (also note that we're -- still -- in the process of moving Pydev things to Aptana, so, hopefully as soon as that's done, no one else will have any problems accessing the Pydev site).

A note for those expecting bugfixes: The Pydev builds now contain not only the current version, but also the repository id in the build (e.g.: 1.4.3.2123), so, if the svn version for a fix is known (I usually post it when I fix a bug), you can wait for the build to get to that version and then get it ( the current nightly build version can be verified at http://nightly.aptana.com/pydev/site.xml and http://nightly.aptana.com/pydev-pro/site.xml )