Refactoring / AST rewriter in Pydev

Right now, there's a bit of quietness in the Pydev realm... I'm working on the current refactoring engine -- which is right now based on the PEPTIC refactoring engine, BRM and the previous Pydev Extension code.

The idea is to stop using BRM altogether and use the others to make up an improved refactoring engine (as that's a big task, I'm avoiding pushing it until it's not stable).

I decided that the first step to do that would be getting the latest PEPTIC code and making it work with the current tests (I had already done some changes to the current Pydev version, so, it diverged a bit).

So, while starting to fix those tests, I realized that its AST rewriter did not properly support the Python 3.0 grammar, so, this is where I am right now, making a decent rewriter. There are 2 AST-based rewriters right now, one that needs the parsing tokens together with the AST and one that only needs the AST -- needles to say none works the way I think they should, so, I decided to go on a different approach, creating a new rewriter.

This rewriter will work with the tokens if they are available (which will make it able to maintain the comments in the correct places) and will generate tokens if they're not available.

I've decided on the following approach: instead of just parsing and writing things to a string in a single step (as the current rewriters do), it'll traverse the structure and fill an auxiliary document structure with the meta information found (i.e.: token requests at a given position and the actual tokens found -- if available -- plus information on the AST, such as statement start and end, nodes, etc), and in a 2nd step, it will generate the actual python code by traversing this document structure. A problem in the previous approaches is that sometimes it found a token and then saw that a comment should be added before it, or a token would not be in the expected AST (and it was really difficult to go back to fix it) -- so, I hope this won't be a problem anymore :)

Unfortunately, as I said, this is a big task, so, I don't think I'll be able to make a release this month -- but looking on the other way, the current release seems pretty stable :)

Git (and Folder compare with Git with WinMerge)

[Update] there's a new post with instructions for configuring git at: http://pydev.blogspot.com/2011/02/git-on-command-line.html

In case it was missed, Pydev is now on Git :)

I'm finding it the hard way that unfortunately its eclipse integration is still not ready for prime time -- it's missing some basic things such as a synchronize view, and I've had some bugs for it to mark the changed files in the package explorer (so, I'm waiting until it matures a bit to give it another try).

In the meanwhile, I'm using the command line (with the msysgit bash).

The thing I missed most was the synchronize view. A part of my modus operandi is making a review of all the changes before doing a commit -- and editing some things there, so, based on this link, I've created a script -- which I called gitdd -- that will create a directory structure with the HEAD and links to the changed files so that one can change those files in a diff viewer (in this case, WinMerge).

The script is below if anyone wants to use it (note that it must be run in the msysgit bash -- although it should probably work on Linux too, with another diff viewer)


#!/bin/sh -e

# usage: gitdd <path>
# Compares the current differences in winmerge with links to original files.
# Note that it must be executed at the root of the git repository.

SUBDIRECTORY_OK=1

O=".git-winmerge-tmp-$$"
V=HEAD
list="$O/list"
list_exist="$O/list_exist"
trap "rm -rf $O" 0
mkdir $O
mkdir $O/b

git diff $V --name-only -z $1 > $list

/d/bin/Python261/python.exe /c/Program\ Files/Git/bin/gitddpy.py $list $O/b/

cat $list | xargs -0 git archive --prefix=a/ $V | /d/bin/bin-1.0/tar.exe xf - -C $O

/d/bin/WinMerge/WinMergeU.exe //r //u //wr //dl WorkingCopy $O/b $O/a


[Update]
I wanted to create links to the files (so that I could edit them on WinMerge and have the original changed), so, I ended up creating a gitddpy.py to create the link (not sure why git prompt was crashing when using the shell for that -- so, this was the solution that worked for me).

The gitddpy.py contents are:


import sys
print sys.argv

f = file(sys.argv[1])
contents = f.read()
import subprocess
import os.path
for line in contents.split('\0'):
    if os.path.exists(line):
        subprocess.call(['cp', line, '--parents', '-l',
        '--target-directory='+sys.argv[2]], shell=True)
f.close()



[Update]
Configuring Git with the following:

git config format.pretty "%h %ct %ad %Cgreen%aN%Creset %s"
git config log.date short

The commands I use most (using the msysgit Bash) are:

git status
git commit -a -m "Message"
git push origin master
git checkout <path>
git log -n 6 --format="%ct %Cgreen%aN%Creset %s"
git commit --amend (change last commit message)

And some of the commands I had to discover how to use in the msysgit bash are:

Alt+Space+E+K: mark contents for copy
Insert: paste contents
Alt+Back: erase until whitespace

The last thing worth mentioning is that the Pydev releases don't have such a nice unique id as the one that was available with subversion anymore, and it now contains the commit time (%ct in the log format). It's a bit annoying that it's such a big number, but I believe that the other choices were worse (the hash would seem totally random for someone seeing it, and basing it on the number of commits wouldn't work with history rewriting).

To sum things up: yes, I know that not being able to do everything in Eclipse sucks a bit, but hopefully the Git tools for Eclipse will catch up soon -- I'm hoping that having the history offline and not having to worry about renames, as well as the whole idea of decentralized versioning systems pays for this nuisance for now (and I'm certain that when the Eclipse plugin for Git catches up it'll be much better).

Pydev 1.5.0 (Pydev Extensions Open Sourced)

Today, Aptana is proud to announce that Pydev and Pydev Extensions have become a single plugin, with all the available contents open source (and freely available for anyone) in the 1.5.0 release (it's the same as 1.4.8 but with all the code open source).

With that, Aptana believes in providing a better service and growth path for Pydev (which will still be actively maintained by Aptana), enabling anyone to provide contributions to the previously closed source product, while providing its Cloud customers a better service.

Note for those already using Pydev or Pydev Extensions:

The update site has been changed (see: http://www.pydev.org/download.html for more details) and if you had a previous install of Pydev Extensions, you need to uninstall it before installing the new version of Pydev.

Note for developers:

Pydev is now available under git (at github), and its previously used subversion will be disabled. Instructions on getting the source code from the new location is available at: http://www.pydev.org/developers.html