Wednesday, December 09, 2009

Pydev 1.5.2 and 1.5.3 released

Just to note, Pydev 1.5.3 was a 1 bug fix release, fixing an annoying bug (an error would be print to the console when running on python -- although everything would still work as expected).

As for 1.5.2, the changes were focused a lot on memory and performance improvements (special thanks to the Eclipse Memory Analyzer, it really helped a lot).

Basically, Pydev had some memory peaks when analyzing modules, because it expected Eclipse to properly manage Jobs created (which are basically threads managed by Eclipse), but it turns out things were not so straightforward, so, say that you changed 500 files (in a source-format operation), it would happily go on and try to create 500 threads in the analysis (that's now fixed).

Other memory savers were targeted at handling Strings (which are 'sort of' internalized now when parsing the code, but without using PermGen space). And not keeping references to Tokens from the parsing alive (because any reference to any Token would keep all alive).

On the performance side, some changes were done to the grammar to make it a bit faster (that area had already been profiled a lot, so, there are not many performance improvement opportunities there).

Aside from that, the compare editor now has proper syntax highlighting, indentation handling and code completion from the Pydev editor. And as usual, some other minor niceties and bug fixes are available.

Tuesday, November 24, 2009

PyQt4 bug preventing it from working properly in Pydev (and workaround)

Hi All,

There have been some reports on people with problems to get PyQt4 working properly in Pydev... This is because depending on the version of PyQt4 it has a major bug that will crash a Python interpreter (this was confirmed in version 4.5.4 -- haven't checked on other versions).

To know if you have a version with this problem you can check it by running the code below in an interactive session:

>>> from PyQt4.QtCore import *
>>> QSignalMapper

>>> import inspect
>>> inspect.ismodule(QSignalMapper) <-- this will make the interpreter
crash!

If that's your problem, right now you have to fix Pydev by editing

eclipse/plugins/org.python.pydev/PySrc/importsTipper.py at line 154,
changing:
"for d in dirComps:"

to:

for d in dirComps:
    if d == 'QSignalMapper':
        continue

Or get another PyQt4 version.

Thursday, November 19, 2009

Pydev 1.5.1 released

Yeap, this release took more time than usually, but I hope it was worth it. The major things as I posted previously were related to the refactoring engine.

There were also a number of other minor problems and features done along the way.

The complete info may be found at: http://pydev.org/

So, now that it's out, what's next?

Well, the last releases were all about giving more power to the user (refactoring, jump in the debugger, app engine, merging pydev extensions to pydev, IronPython support -- did I mention that the pydev debugger is now working with the latest IronPython?), so, I think that now would be a good time to step back a little and work on ironing out the bugs and making some profiling sessions (for both speed and memory).

Wednesday, November 11, 2009

Pydev 1.5.1 almost there

Ok, all the planned changes for 1.5.1 are now in place and the current version already seems pretty stable, so, the current nightly build (available as an update site at http://pydev.org/nightly) will be the final version unless there's some critical bug found (so, if there's anyone interested, it'd be nice testing that version).

The major changes were:

- Improvements in the AST rewriter
- Improvements on the refactoring engine (no longer using BRM, merged with the latest PEPTIC and added/fixed lots of cases for corner-case situations)
- Improvement in the type inference engine for lists

And along that there were a good number of minor editor improvements and bug-fixes.

Thursday, October 29, 2009

New Pydev nightly build

The new nightly build for Pydev is now http://pydev.org/nightly (just add that to the eclipse update manager).

The current nightly build already contains the changes to the refactoring engine... If anyone wants to get it and test it, now would be a good time :)

Pydev refactoring engine

Ok, right now, the git version of Pydev already has the changes in the refactoring engine incorporated... So, right now, it's all based on the previous pydev extensions rename and PEPTIC for the other refactorings (which also had a number of bugs fixed).

Now what's missing for the next release is some bug-fixing (in other areas of the code).

Tuesday, September 29, 2009

Refactoring / AST rewriter in Pydev

Right now, there's a bit of quietness in the Pydev realm... I'm working on the current refactoring engine -- which is right now based on the PEPTIC refactoring engine, BRM and the previous Pydev Extension code.

The idea is to stop using BRM altogether and use the others to make up an improved refactoring engine (as that's a big task, I'm avoiding pushing it until it's not stable).

I decided that the first step to do that would be getting the latest PEPTIC code and making it work with the current tests (I had already done some changes to the current Pydev version, so, it diverged a bit).

So, while starting to fix those tests, I realized that its AST rewriter did not properly support the Python 3.0 grammar, so, this is where I am right now, making a decent rewriter. There are 2 AST-based rewriters right now, one that needs the parsing tokens together with the AST and one that only needs the AST -- needles to say none works the way I think they should, so, I decided to go on a different approach, creating a new rewriter.

This rewriter will work with the tokens if they are available (which will make it able to maintain the comments in the correct places) and will generate tokens if they're not available.

I've decided on the following approach: instead of just parsing and writing things to a string in a single step (as the current rewriters do), it'll traverse the structure and fill an auxiliary document structure with the meta information found (i.e.: token requests at a given position and the actual tokens found -- if available -- plus information on the AST, such as statement start and end, nodes, etc), and in a 2nd step, it will generate the actual python code by traversing this document structure. A problem in the previous approaches is that sometimes it found a token and then saw that a comment should be added before it, or a token would not be in the expected AST (and it was really difficult to go back to fix it) -- so, I hope this won't be a problem anymore :)

Unfortunately, as I said, this is a big task, so, I don't think I'll be able to make a release this month -- but looking on the other way, the current release seems pretty stable :)

Tuesday, September 08, 2009

Git (and Folder compare with Git with WinMerge)

[Update] there's a new post with instructions for configuring git at: http://pydev.blogspot.com/2011/02/git-on-command-line.html

In case it was missed, Pydev is now on Git :)

I'm finding it the hard way that unfortunately its eclipse integration is still not ready for prime time -- it's missing some basic things such as a synchronize view, and I've had some bugs for it to mark the changed files in the package explorer (so, I'm waiting until it matures a bit to give it another try).

In the meanwhile, I'm using the command line (with the msysgit bash).

The thing I missed most was the synchronize view. A part of my modus operandi is making a review of all the changes before doing a commit -- and editing some things there, so, based on this link, I've created a script -- which I called gitdd -- that will create a directory structure with the HEAD and links to the changed files so that one can change those files in a diff viewer (in this case, WinMerge).

The script is below if anyone wants to use it (note that it must be run in the msysgit bash -- although it should probably work on Linux too, with another diff viewer)


#!/bin/sh -e

# usage: gitdd <path>
# Compares the current differences in winmerge with links to original files.
# Note that it must be executed at the root of the git repository.

SUBDIRECTORY_OK=1

O=".git-winmerge-tmp-$$"
V=HEAD
list="$O/list"
list_exist="$O/list_exist"
trap "rm -rf $O" 0
mkdir $O
mkdir $O/b

git diff $V --name-only -z $1 > $list

/d/bin/Python261/python.exe /c/Program\ Files/Git/bin/gitddpy.py $list $O/b/

cat $list | xargs -0 git archive --prefix=a/ $V | /d/bin/bin-1.0/tar.exe xf - -C $O

/d/bin/WinMerge/WinMergeU.exe //r //u //wr //dl WorkingCopy $O/b $O/a


[Update]
I wanted to create links to the files (so that I could edit them on WinMerge and have the original changed), so, I ended up creating a gitddpy.py to create the link (not sure why git prompt was crashing when using the shell for that -- so, this was the solution that worked for me).

The gitddpy.py contents are:


import sys
print sys.argv

f = file(sys.argv[1])
contents = f.read()
import subprocess
import os.path
for line in contents.split('\0'):
    if os.path.exists(line):
        subprocess.call(['cp', line, '--parents', '-l',
        '--target-directory='+sys.argv[2]], shell=True)
f.close()



[Update]
Configuring Git with the following:

git config format.pretty "%h %ct %ad %Cgreen%aN%Creset %s"
git config log.date short

The commands I use most (using the msysgit Bash) are:

git status
git commit -a -m "Message"
git push origin master
git checkout <path>
git log -n 6 --format="%ct %Cgreen%aN%Creset %s"
git commit --amend (change last commit message)

And some of the commands I had to discover how to use in the msysgit bash are:

Alt+Space+E+K: mark contents for copy
Insert: paste contents
Alt+Back: erase until whitespace

The last thing worth mentioning is that the Pydev releases don't have such a nice unique id as the one that was available with subversion anymore, and it now contains the commit time (%ct in the log format). It's a bit annoying that it's such a big number, but I believe that the other choices were worse (the hash would seem totally random for someone seeing it, and basing it on the number of commits wouldn't work with history rewriting).

To sum things up: yes, I know that not being able to do everything in Eclipse sucks a bit, but hopefully the Git tools for Eclipse will catch up soon -- I'm hoping that having the history offline and not having to worry about renames, as well as the whole idea of decentralized versioning systems pays for this nuisance for now (and I'm certain that when the Eclipse plugin for Git catches up it'll be much better).

Thursday, September 03, 2009

Pydev 1.5.0 (Pydev Extensions Open Sourced)

Today, Aptana is proud to announce that Pydev and Pydev Extensions have become a single plugin, with all the available contents open source (and freely available for anyone) in the 1.5.0 release (it's the same as 1.4.8 but with all the code open source).

With that, Aptana believes in providing a better service and growth path for Pydev (which will still be actively maintained by Aptana), enabling anyone to provide contributions to the previously closed source product, while providing its Cloud customers a better service.

Note for those already using Pydev or Pydev Extensions:

The update site has been changed (see: http://www.pydev.org/download.html for more details) and if you had a previous install of Pydev Extensions, you need to uninstall it before installing the new version of Pydev.

Note for developers:

Pydev is now available under git (at github), and its previously used subversion will be disabled. Instructions on getting the source code from the new location is available at: http://www.pydev.org/developers.html

Friday, August 21, 2009

Pydev 1.4.8 released & Live coding in Pydev

Pydev 1.4.8 has been released (a few days ago).

The usability improvements on this release had already been highlighted before, so, I think that the major highlight to talk about is the possibility of 'live coding' in the debugger (although it's still a bit on the experimental side).

To enable that, 2 features were done: Jump to line (ctrl+R with the focus on the line) and reload module contents (right now only available for python 2.x).

The idea is that you can get to some point in your code in the debugger, watch what's happening, change the code to fix it and then jump to the line that created that frame (so, you actually need to do a jump to an outer frame -- which actually requires finishing the current frame, as python has no support for dropping a frame -- although you can jump to the return line and then jump in the outer frame to the caller of that frame -- yes, it can be a bit confusing).

Note that there are still a number of shortcomings in the current version -- some because of python, which can only make a jump when the debugger receives a line event (so, if you were at the end of the frame, it might be that you cannot properly do the jump, because a return event will be available in that case, instead of a line event, and being inside a try..except has its peculiarities too), and others because the reload is still pretty naive (it uses the xreload module -- currently only available for 2.x -- but I already have some ideas that can make it better).

So, in the current version, it can be a bit difficult to use it properly, but still, it's pretty nice when it does :)

p.s.: I was going to provide a screencast for that, but decided on waiting a bit more until I'm able to work on some of the rough edges in the current implementation.

p.s2.: The jump to line in python can only jump inside the current topmost frame, but in pydev you can ask to make a jump to any place in the stack, but that means it'll execute everything normally until it gets to the frame where the jump was asked -- and only then will it actually execute the jump -- maybe if there's enough interest in that, someone at the python side could provide a way to drop frames :)

Monday, August 03, 2009

Twitter, Pydev homepage & Usability improvements

I've recently started to Tweet to reporting on what's happening in Pydev (http://twitter.com/fabiofz), so, it might be worth for those that want to get on the bits & pieces around the unreleased version.

Now, on with what's happening in the Pydev world: the Pydev and the Pydev Extensions homepages were updated -- yes, they were outdated in a number of things -- Pydev is evolving too fast for them to keep up :)

It was pretty nice that while going over the editors/details to explain them on the homepage, I ended up reorganizing things and improving the usability as a direct result of that (I think it's much easier/better to code and give a better interface than to explain why something doesn't actually work some way in minor details).

So, hopefully those usability improvements will make the lives easier for people using Pydev (and for those starting up too).

I've already published the homepage with screeshots of those improvements, but right now, they're only available in the nightly build (and will be released at 1.4.8)

Some of those usability improvements are:
  • Preview of the color options chosen
  • Preview of the code formatting chosen
  • Preview of changing the code style
  • Preview of block comments
  • Auto imports properly working when the editor changes (ctrl+shift+O in Pydev Extensions)
  • Hover works on markers when the editor changes
  • A Pydev (Jython) project can coexist with a JDT project (and properly use its info -- only project references worked previously)
  • Improved error messages for Iron Python not being version 2.6
  • The interactive console no longer halts the UI thread
... and some others I don't remember anymore ...

Some rough edges are probably still lurking around, but hopefully it has become harder to spot those :)

Saturday, July 25, 2009

Creating a dark theme (and exporting and importing Eclipse preferences)

Usually I use my colors with the default settings that Eclipse provides, but I know many people that like alternative appearances for Eclipse (usually using a dark theme), so, I decided to give it a try...

For changing the default colors, there are some places that you have to look through in the window > preferences menu:
  • General > Editors > Text Editors (to set the background color / selection color, etc. for all editors)
  • General > Editors > Text Editors > Annotations (to set the colors for the occurrences highlight, tasks, etc.)
  • Pydev > Editor (to set the pydev editor-related colors)
  • General > Appearance > Colors an Fonts (to set what's not available in the other places)
Now, after you've properly set those colors, one thing you should really do is export the preferences you've just changed so that you can import them later.

To do that, go to File > Export > Preferences and export all the preferences to a file, then, open that file and remove all the non-color related preferences (except the file_export_version, which Eclipse requires when doing the import) -- note that you can keep other preferences you enter all the time too, and it might be easier doing it with a clean Eclipse install just with what you want to config (so that you don't have too many preferences in that preferences file).

Later, you can import those preferences in Eclipse through File > Import > Preferences.

The preferences for the dark theme I'm using gives the result below (note that you may have to restart Eclipse so that the left ruler looks correct):



This can be reproduced by saving the contents below in a 'dark_theme.epf' file (the extension stands for eclipse preferences file) and importing it.


file_export_version=3.0
/instance/org.python.pydev/BACKQUOTES_COLOR=255,255,255
/instance/org.python.pydev/COMMENT_COLOR=153,51,204
/instance/org.python.pydev/CODE_COLOR=255,255,255
/instance/org.python.pydev/NUMBER_COLOR=204,255,51
/instance/org.python.pydev/CLASS_NAME_COLOR=255,204,0
/instance/org.python.pydev/KEYWORD_COLOR=255,102,0
/instance/org.python.pydev/SELF_COLOR=255,255,0
/instance/org.python.pydev/FUNC_NAME_COLOR=255,204,0
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionForeground.SystemDefault=false
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Background.SystemDefault=false
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Background=0,0,0
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Foreground.SystemDefault=false
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Foreground=255,255,255
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionBackground.SystemDefault=false
/instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionBackground=0,0,136
/instance/org.eclipse.ui.editors/pydevOccurrenceIndicationColor=128,64,0
/instance/org.eclipse.ui.editors/lineNumberColor=255,255,255
/instance/org.eclipse.ui.editors/printMargin=true
/instance/org.eclipse.ui.editors/printMarginColor=255,0,0
/instance/org.eclipse.ui.editors/currentLineColor=70,70,70
/instance/org.eclipse.ui.editors/currentIPTextStyle=BOX
/instance/org.eclipse.ui.editors/currentIPIndication=true
/instance/org.eclipse.ui.editors/currentIPHighlight=false
/instance/org.eclipse.ui.editors/secondaryIPTextStyle=DASHED_BOX
/instance/org.eclipse.ui.editors/secondaryIPHighlight=false
/instance/org.eclipse.ui.editors/secondaryIPIndication=true



Update: Added the debug call stack and debug current line annotations to appear as dashed box and box.

Wednesday, July 08, 2009

Pydev 1.4.7 released

Pydev 1.4.7 is now available for download.

Aside from the regular update site, there are new mirrors for getting pydev/pydev extensions, so the update site mirrors are available as follows:

http://fabioz.com/pydev/updates (contains both, pydev and pydev extensions)
http://pydev.sf.net/updates (contains only pydev)
http://update.aptana.com/update/pydev/3.2 (contains only pydev)
http://update.aptana.com/update/pydev-pro/3.2 (contains only pydev extensions)

The major features (Iron Python integration and the new way of running as unit-test / running the current editor with the configured nature) were previously discussed in the following post: http://pydev.blogspot.com/2009/06/testing-on-pydev-147-ironpython.html and are now available in this release.

Note that the Iron Python debugger is still not available in this release, but it seems that the next release of Iron Python will have the sys.settrace() support, so, there's a chance that the next Pydev version has a debugger for it.

Aside from those features, other noteworthy things in this release are:
  • the debug console can be used with the remote debugger (pydev extensions)
  • the auto-config was fixed
  • context-independent completions are now offered when the type inference is not able to get context-dependent completions on a given token (pydev extensions)
  • the google app engine integration had some fixes

Sunday, June 28, 2009

Testing on Pydev 1.4.7 & IronPython

Pydev 1.4.7 is now ready to be tested for a new release (watch for the nightly build with svn version 2834 -- http://pydev.blogspot.com/2009/02/pydev-nightly-builds.html )

One of the major features in this release is the IronPython integration. Note that it requires IronPython 2.6 (earlier versions are not supported because of the lack of sys._getframe()).

In that integration, code-completion should be working for the .NET libraries and other dlls. For the .NET libraries, just make sure that the library you're using is listed in the forced builtins ( http://fabioz.com/pydev/manual_101_interpreter.html ) -- I believe all libraries should be there already, but it could be that some default library is missing.

To use code-completion for other libraries, the code must be compiled as a dll (because Pydev does not recognize C# nor VB) and that dll must have the exact same name of the namespace it contains (e.g.: if you want to do "import iTextSharp", the dll must be named iTextSharp.dll -- the case of the file important there) and the folder containing that dll must be added as a source folder (or external source folder) -- which is the same thing for python compiled extensions.

Most features should be working fine with IronPython, with the most notable exception being that the debugger is not supported (that's because in this version, IronPython still doesn't provide sys.settrace(), so, there's no API to work on for python debugging).

Other niceties were also added in this release:

  • Ctrl+F9 (which was previously a run as jython) now executes a run as unit-test based on the project type, providing an interface where the user can filter which tests should be run, and F9 which was a run-as python is now a run as current project type -- so, it works for python, jython and ironpython projects).

  • Pydev Extensions context-insensitive completions are shown for any token which cannot be determined by the type inference engine (previously it did that only for method parameters) -- and the number of chars for showing those completions can be customized (by default, 2 chars are required).

Wednesday, May 20, 2009

Pydev 1.4.6 released / Google App Engine on Pydev video

Pydev 1.4.6 has just been released.

The major highlights in this release are:

The project and interpreter configuration was improved:
  • The pydev package explorer will now show problems in the project configuration (until now, those misconfigurations wouldn't be properly shown and some subtle problems could arise because of those misconfigurations);
  • The project pythonpath can use variables;
  • The interpreter can be referred from a user-given identification.

There is an easier way to create and manage a Google App Engine project in Pydev. The video below shows how to properly config/upload/manage it from Pydev:

Tuesday, May 12, 2009

Testing on Pydev 1.4.6 & Google App Engine Integration

Ok, Pydev 1.4.6 is mostly finished, just waiting for some more tests before the final release... those who'd like to get it and test it are very welcome to do so, as it should be pretty stable already (see: http://pydev.blogspot.com/2009/02/pydev-nightly-builds.html on how to get it).

One of the major features in this release is targeted at people using Google App Engine in Pydev: now there's specific support for creating, running and managing/uploading a Google App Engine project.

To create a new project, there's a new wizard:


The specific parts of the wizard are:

Location of the Google App Engine installation directory

From there it gets the settings needed to create a GOOGLE_APP_ENGINE string substitution variable (that's also something new in 1.4.6: you can configure the PYTHONPATH of your project using variables) and the paths for the external libraries to be added to the PYTHONPATH of the new project.



Choose a template to start the project

Do you want a different quick start? If you think there's a common project structure that's missing, please send me a zip of the project with that structure to be added to the plugin (with a brief description for it).




After that, with the project created, you can use the Pydev Google App Engine specific actions by right-clicking the folder that has the app.yaml file (see number 1 in the image). From there you can launch it (number 2) or manage it (number 3).




To upload your application to Google App Engine, open the dialog below (from choosing "Pydev: Manage project in Google App Engine") and click 'Run' (or choose/enter another command). If a username or password or any other input is requested, enter it in the text-field and press 'Enter command' (it should properly echo the password as '*' -- if there's some situation where it's not echoing the char and it should, please enter a bug report).


It's also interesting to note that any existing project can be configured as a Pydev Google App Engine project (thus having those actions) by specifying a GOOGLE_APP_ENGINE string substitution variable in the Pydev Project properties -- and don't forget that the actions should be executed on the directory that contains your app.yaml file.

Sunday, April 12, 2009

Pydev 1.4.5 released

First, I'd like to thank those that took the time to test Pydev before the official release (some bugs were fixed because of those reports).

This release had quite some work on the Python grammar error handling. The major difference for Pydev users is that operations that require a valid AST to be performed should be more reliable -- and a bit faster -- when syntax errors are present in the code (e.g.: showing the outline; requesting a code-completion -- previously some text-manipulations were performed to make a valid AST, but this was not very reliable and also required more resources, as it tried to reparse the whole buffer after each text manipulation).

The code formatter also had some work. Particularly, four requests that popped up now and then are now available:

- A context menu is provided for folders so that all files under it can be (recursively) formatted
- The code-formatting can be automatically applied when saving the editor
- A new line can be added at the end of files
- Whitespaces can be trimmed

One other area I believe deserves some attention is the Pydev Package Explorer, which was improved so that working sets can be shown as top-level items and folders without '__init__.py' have a regular folder icon (instead of a package icon).

And as regularly, bug-fixes and other minor features were made available (see the Pydev homepage for more details)

Saturday, April 04, 2009

Extended VS Presentation

Over the last days I've been trying a new skin to Eclipse:

http://code.google.com/p/skin4eclipse

and it's pretty nice. It did have some minor quirks, like not being able to use ctrl+page up/down to traverse the editors and some other minors, but as it's open source, I was able to add the things I was missing -- which have already been incorporated by Andrei, who is the author of the plugin (those were changed when the current release was 1.5.1, so, it'll be only available in a later release).

It's pretty nice that it keeps track of the editors that were recently closed (so, you can reopen them) and it maintains the order of the tabs when the editors are closed (a.k.a: MRU bug). Also, it uses less space for the editor name, so, you can see more opened tabs at the same time.

The colors and fonts can also be configured (for me the default colors drew too much attention to the titlebar).

The only thing I still wasn't able to figure is if the presentation can prevent detached tabs to be closed with ESC (that's something that really bothers me, because I usually leave a detached tab for the console in a separate monitor, and now and then those end up being closed accidentally with ESC) -- but the default presentation has the same issue (I'm not even sure if there's a way around that).

Update: I've just sent a patch to add an option to prevent ESC from closing detached editors :)

Monday, March 30, 2009

New Pydev release soon (call for testers)

A new Pydev release should be available pretty soon... if no critical bugs are found, the current nightly build -- version 1.4.5.2712 -- will be the released version.

So, whoever is interested in the next release, please get the latest nightly build -- http://pydev.blogspot.com/2009/02/pydev-nightly-builds.html -- and check it to see if everything is working fine (so that I can provide fixes for any critical bug found before it's actually officially released).

Note that this is an attempt to have released versions more stable -- because even with a good test suite and some days of manual testing before an actual release, now and then there are still critical bugs slipping (there are way too many different scenarios and use-cases in which Pydev is used)

Sure, it's nothing like the 1-2 months of testing Eclipse goes through before a release... Actually, from submitting something up to approval in Eclipse, it usually means 6-10 months for it to be in a final release... for instance: I've just submitted a rather minor patch to make the word completion (Emacs-Style) faster -- but it'll only be patched for Eclipse 3.6... meaning that with luck it'll be available 10 months from now (because it's in API freeze now and in 2 days will stay at feature freeze and will get its 1-2 months of testing) -- but on the bright side, I can patch my own version :)

Tuesday, March 03, 2009

Pydev zip distribution problem

If you've installed Pydev with the zip files, you are probably having problems with code-completion. This happened because in the process of updating pydev to P2 (which is the new way of installing eclipse plugins), the zip files ended up having the update site and not the plugins in the format they should be installed.

You can fix that by uninstalling pydev and then instead of making any extract, go to help > software updates > available software > add site > archive and add each zip file there (org.python.pydev.feature-1.4.4.2636.zip and com.python.pydev.extensions-1.4.4.2636.zip), which should use those files correctly as an update site.

The error that appears in the error log if that's not done is something like:
Error creating python process - exited before creating sockets -
exitValue = (1)(/usr/bin/python2.4
/tmp/eclipse-config/org.eclipse.osgi/bundles/650/1/.cp/PySrc/pycompletionse
rver.py)

Tuesday, February 24, 2009

Pydev 1.4.4

Just now I realized (thanks to Radim) that I forgot to post about 1.4.4 (which was released some hours after 1.4.3).

Pydev 1.4.4 was a one bugfix release: it fixes a critical bug when configuring the interpreter (if no environment variables were specified, it was not possible to configure an interpreter).

So, if you got 1.4.3, please upgrade to 1.4.4! (1.4.3 was removed from the update site because of that bug).

Pydev, JavaCC and error recovery

Pydev so far had a very primitive way of trying to recover from errors: it tried to change the document to make it 'right', with some really simple text-based heuristics, which were effective for some common cases. Still that's really suboptimal for a number of cases, and, error recovery while parsing is the best approach for dealing with that.

With that in mind, error recovery is being added to the Pydev JavaCC grammar. I must say that one of the largest problems to add it is the lack of documentation on the subject and what's the most effective way to deal with it.

Also, JavaCC has no support for bactracking while parsing (it can lookahead to search which path to take, but after a path is taken, it cannot go back to change it), and if at sometime it cannot find a path to take or if some token in the path was not found, a ParseException is raised.

The docs I found about how to handle errors would point that the correct way of treating ParseExceptions is catching those and trying to revert to a stable state in the parsing machine, but I found one other interesting article on the subject Adding Automatic Syntax Error Repair to a Java-based Parser Generator (from Pieter van der Spek, Nico Plat and Kees Pronk) , where it explains how could auto-recovery be added to JavaCC.

In the end, looking in the Pydev grammar, I found one interesting property: as part of the effort to make pretty-printing of the Pydev AST, lots of tokens were being looked for and added as 'special tokens' to the nodes created.

i.e.: it'd go and look for a colon token and add it as a special token to the node before it -- and if not found, an error would be thrown at that point.

Now, I ended up extending that approach so that if it went to look for a token and didn't find it, the error for not finding it is reported, but not thrown as an exception. Instead, it goes on to create that token to be consumed after the current token.

The only other gotcha is that the grammar makes a skip for new lines, and it happens 'under the hood' for the grammar (this is a really tricky area), so, in the end the grammar had to be changed so that it would handle the suite() construct even if some indentation 'disappeared' -- but it only goes that path if there is a problem in the syntax, so, that's reported as an error.

In the end I mixed both approaches. A part goes for the 'preemptive' attempt to create tokens when we know it should be there and it's not and another handles the ParseExceptions to try to recover from it.

The preemptive constructs look like:

{this.findTokenAndAdd(":");} <COLON>, where findTokenAndAdd reports the error and creates the token if it's not found

And the others work around the ParseExceptions:

E.g.:

Token Name() #Name:{Token t;}
{
    try{
        t =
    }catch(ParseException e){
        t = handleErrorInName(e);
    }
    { ((Name)jjtThis).id = t.image; return t; } {}
}

Thursday, February 19, 2009

It's out: Pydev 1.4.3

This release did have some nice features added, such as:
  • Environment variables can be set per interpreter (not having this was nasty because it forced you to have multiple workspaces running from pre-customized shells if you did require different environment variables for each of your interpreters).
  • A template can be chosen when creating a new module
  • The interactive console can start with any of the available interpreters.
  • Ctrl + 2 + --reindex can be used to recreate the internal indexes of Pydev -- this was added because of a bug -- which is described below and fixed, so, hopefully this won't be needed by anyone :)
But it focused a lot more on bug-fixing (more than 20 bugs were fixed), with a special note for:
  • The bug which made the cursor 'jump' because of some spurious selection events from the outline in Linux (so, when typing it could jump to the place you just edited and you'd start overriding it) -- special thanks to Mike Auty for providing the info needed to fix it.
  • Some racing conditions could corrupt the Pydev indexing (the effect was that the globals browser and context-insensitive code-completion would sometimes not find some tokens that were supossed to be there).
Enjoy!

p.s.: The main update urls now only contain versions 1.4.3 and onwards (those are now using the new p2 update site structure, so, please check http://fabioz.com/pydev/download.html and http://pydev.sourceforge.net/download.html to see how to obtain some earlier version -- if you ever need it)

Friday, February 06, 2009

Pydev Nightly builds

Yeap, it's available for those that really want to keep up with the most up-to-date code...

The nightly build is available in the format of an update site.

The update site links to use are:

http://nightly.aptana.com/pydev/site.xml
http://nightly.aptana.com/pydev-pro/site.xml

Instructions on how to use it are available at:

http://nightly.aptana.com/pydev/index.html
http://nightly.aptana.com/pydev-pro/index.html

Until now, Pydev didn't have an easy way for people to get updates until a proper release was all wrapped up, so, this should be pretty handy for those of you expecting a bugfix -- or just wanting to experiment with the latest thing you're waiting for. Also, usually before committing I do run the Pydev test-suite (which is pretty big), so, it should be regularly pretty stable.

Another thing: I've been having some complains on people not being able to access fabioz.com, even with the server being up (it appears it's related to some specific providers), so, that update site can be used as a second choice if you're not currently able to access fabioz.com to get Pydev Extensions (also note that we're -- still -- in the process of moving Pydev things to Aptana, so, hopefully as soon as that's done, no one else will have any problems accessing the Pydev site).

A note for those expecting bugfixes: The Pydev builds now contain not only the current version, but also the repository id in the build (e.g.: 1.4.3.2123), so, if the svn version for a fix is known (I usually post it when I fix a bug), you can wait for the build to get to that version and then get it ( the current nightly build version can be verified at http://nightly.aptana.com/pydev/site.xml and http://nightly.aptana.com/pydev-pro/site.xml )

Monday, January 26, 2009

New Pydev repository location

Hi All,

The new subversion location for Pydev is now:

http://svn.aptana.com/svn/pydev/

Along with that change, the actual structure of things changed to conform to the other projects at Aptana (Pydev had flat structure containing all the plugins in the root level, and now it has a structure with 'features', 'plugins' and 'builders' and each plugin is in one of those folders).

Tuesday, January 13, 2009

Pydev 1.4.2

Ok, the Pydev 1.4.2 is just out.

The major feature this release has been long awaited: each project can now have a different configured interpreter.

As a result of that change, some internal structures will have to be recreated. So, right after upgrading it's expected that Pydev will do some processing to make that right, and now, it has to gather the information for all the interpreters, and not only for the current one, so, be a bit patient while it does that (it'll do that just once).

Also, many fixes were made available, so, upgrading is really reccommended.