Tuesday, September 29, 2009

Refactoring / AST rewriter in Pydev

Right now, there's a bit of quietness in the Pydev realm... I'm working on the current refactoring engine -- which is right now based on the PEPTIC refactoring engine, BRM and the previous Pydev Extension code.

The idea is to stop using BRM altogether and use the others to make up an improved refactoring engine (as that's a big task, I'm avoiding pushing it until it's not stable).

I decided that the first step to do that would be getting the latest PEPTIC code and making it work with the current tests (I had already done some changes to the current Pydev version, so, it diverged a bit).

So, while starting to fix those tests, I realized that its AST rewriter did not properly support the Python 3.0 grammar, so, this is where I am right now, making a decent rewriter. There are 2 AST-based rewriters right now, one that needs the parsing tokens together with the AST and one that only needs the AST -- needles to say none works the way I think they should, so, I decided to go on a different approach, creating a new rewriter.

This rewriter will work with the tokens if they are available (which will make it able to maintain the comments in the correct places) and will generate tokens if they're not available.

I've decided on the following approach: instead of just parsing and writing things to a string in a single step (as the current rewriters do), it'll traverse the structure and fill an auxiliary document structure with the meta information found (i.e.: token requests at a given position and the actual tokens found -- if available -- plus information on the AST, such as statement start and end, nodes, etc), and in a 2nd step, it will generate the actual python code by traversing this document structure. A problem in the previous approaches is that sometimes it found a token and then saw that a comment should be added before it, or a token would not be in the expected AST (and it was really difficult to go back to fix it) -- so, I hope this won't be a problem anymore :)

Unfortunately, as I said, this is a big task, so, I don't think I'll be able to make a release this month -- but looking on the other way, the current release seems pretty stable :)

5 comments:

Alexandre Fayolle said...

have you considered astng (an underlying library to pylint)?

http://www.logilab.org/project/astng

Alexandre

Fabio Zadrozny said...

Hi Alexandre,

Although I already knew about it, I haven't considered it because Pydev already does lots of AST manipulations/analysis in its current structure (which uses an improved fork of the Jython AST structure), so, I don't consider it viable to change all that and start using another AST library.

Cheers,

Fabio

httpdss said...

Impressive work fabio!

Im not much of an eclipse plugin developer, but I was Pydev A LOT!

Besides that, i wanted to make a suggestion on your recent move to github... it could be of great help for developers that want to collaborate, be able to see the ongoing branches, leaving "master" as stable as possible (as you say).
This could be done by pushing, for example, your "ast_rewriter" branch to github so people can help you get features out more quickly ;)

Hope this help, and im looking forward to know what you think about it...

Cheers,
Kenny

Tim Diggins said...

Good news about improving refactorings... And by the way since you're on git - no excuse for "not pushing" that's what the super-easy branching and merging is for! (I recommend git-remote-branch / grb for manipulating branches -- explains the command line really well: http://github.com/webmat/git_remote_branch
)

As a side note (maybe I should take this to email - feel free to suggest that) I'd like to have autocompletion for a particular python-based DSL I'm working on. How plausible is it to extend pydev to do that? Can you suggest where I should look first in the code?

(The joys of the whole of pydev being opensource!)

Fabio Zadrozny said...

Hi Tim,

Thanks for the git link (I'll take a look at it).

As for your python-based DLS, it's doable...

I think the 1st step would be creating a parser to create the internal AST structure from your code.

After that, the Pydev AST creation would need to change to use that parser when needed and adding the extension of your files for Pydev to handle -- doing that should give you code completion, and all the current features (but you may still need to create a different editor if the syntax highlighting doesn't work for your case).

p.s. If you have further doubts on that, please ask at the pydev-code list (at sourceforge)