Just now I realized (thanks to Radim) that I forgot to post about 1.4.4 (which was released some hours after 1.4.3).
Pydev 1.4.4 was a one bugfix release: it fixes a critical bug when configuring the interpreter (if no environment variables were specified, it was not possible to configure an interpreter).
So, if you got 1.4.3, please upgrade to 1.4.4! (1.4.3 was removed from the update site because of that bug).
Tuesday, February 24, 2009
Pydev, JavaCC and error recovery
Pydev so far had a very primitive way of trying to recover from errors: it tried to change the document to make it 'right', with some really simple text-based heuristics, which were effective for some common cases. Still that's really suboptimal for a number of cases, and, error recovery while parsing is the best approach for dealing with that.
With that in mind, error recovery is being added to the Pydev JavaCC grammar. I must say that one of the largest problems to add it is the lack of documentation on the subject and what's the most effective way to deal with it.
Also, JavaCC has no support for bactracking while parsing (it can lookahead to search which path to take, but after a path is taken, it cannot go back to change it), and if at sometime it cannot find a path to take or if some token in the path was not found, a ParseException is raised.
The docs I found about how to handle errors would point that the correct way of treating ParseExceptions is catching those and trying to revert to a stable state in the parsing machine, but I found one other interesting article on the subject Adding Automatic Syntax Error Repair to a Java-based Parser Generator (from Pieter van der Spek, Nico Plat and Kees Pronk) , where it explains how could auto-recovery be added to JavaCC.
In the end, looking in the Pydev grammar, I found one interesting property: as part of the effort to make pretty-printing of the Pydev AST, lots of tokens were being looked for and added as 'special tokens' to the nodes created.
i.e.: it'd go and look for a colon token and add it as a special token to the node before it -- and if not found, an error would be thrown at that point.
Now, I ended up extending that approach so that if it went to look for a token and didn't find it, the error for not finding it is reported, but not thrown as an exception. Instead, it goes on to create that token to be consumed after the current token.
The only other gotcha is that the grammar makes a skip for new lines, and it happens 'under the hood' for the grammar (this is a really tricky area), so, in the end the grammar had to be changed so that it would handle the suite() construct even if some indentation 'disappeared' -- but it only goes that path if there is a problem in the syntax, so, that's reported as an error.
In the end I mixed both approaches. A part goes for the 'preemptive' attempt to create tokens when we know it should be there and it's not and another handles the ParseExceptions to try to recover from it.
The preemptive constructs look like:
{this.findTokenAndAdd(":");} <COLON>, where findTokenAndAdd reports the error and creates the token if it's not found
And the others work around the ParseExceptions:
E.g.:
Token Name() #Name:{Token t;}
{
try{
t =
}catch(ParseException e){
t = handleErrorInName(e);
}
{ ((Name)jjtThis).id = t.image; return t; } {}
}
With that in mind, error recovery is being added to the Pydev JavaCC grammar. I must say that one of the largest problems to add it is the lack of documentation on the subject and what's the most effective way to deal with it.
Also, JavaCC has no support for bactracking while parsing (it can lookahead to search which path to take, but after a path is taken, it cannot go back to change it), and if at sometime it cannot find a path to take or if some token in the path was not found, a ParseException is raised.
The docs I found about how to handle errors would point that the correct way of treating ParseExceptions is catching those and trying to revert to a stable state in the parsing machine, but I found one other interesting article on the subject Adding Automatic Syntax Error Repair to a Java-based Parser Generator (from Pieter van der Spek, Nico Plat and Kees Pronk) , where it explains how could auto-recovery be added to JavaCC.
In the end, looking in the Pydev grammar, I found one interesting property: as part of the effort to make pretty-printing of the Pydev AST, lots of tokens were being looked for and added as 'special tokens' to the nodes created.
i.e.: it'd go and look for a colon token and add it as a special token to the node before it -- and if not found, an error would be thrown at that point.
Now, I ended up extending that approach so that if it went to look for a token and didn't find it, the error for not finding it is reported, but not thrown as an exception. Instead, it goes on to create that token to be consumed after the current token.
The only other gotcha is that the grammar makes a skip for new lines, and it happens 'under the hood' for the grammar (this is a really tricky area), so, in the end the grammar had to be changed so that it would handle the suite() construct even if some indentation 'disappeared' -- but it only goes that path if there is a problem in the syntax, so, that's reported as an error.
In the end I mixed both approaches. A part goes for the 'preemptive' attempt to create tokens when we know it should be there and it's not and another handles the ParseExceptions to try to recover from it.
The preemptive constructs look like:
{this.findTokenAndAdd(":");} <COLON>, where findTokenAndAdd reports the error and creates the token if it's not found
And the others work around the ParseExceptions:
E.g.:
Token Name() #Name:{Token t;}
{
try{
t =
}catch(ParseException e){
t = handleErrorInName(e);
}
{ ((Name)jjtThis).id = t.image; return t; } {}
}
Thursday, February 19, 2009
It's out: Pydev 1.4.3
This release did have some nice features added, such as:
p.s.: The main update urls now only contain versions 1.4.3 and onwards (those are now using the new p2 update site structure, so, please check http://fabioz.com/pydev/download.html and http://pydev.sourceforge.net/download.html to see how to obtain some earlier version -- if you ever need it)
- Environment variables can be set per interpreter (not having this was nasty because it forced you to have multiple workspaces running from pre-customized shells if you did require different environment variables for each of your interpreters).
- A template can be chosen when creating a new module
- The interactive console can start with any of the available interpreters.
- Ctrl + 2 + --reindex can be used to recreate the internal indexes of Pydev -- this was added because of a bug -- which is described below and fixed, so, hopefully this won't be needed by anyone :)
- The bug which made the cursor 'jump' because of some spurious selection events from the outline in Linux (so, when typing it could jump to the place you just edited and you'd start overriding it) -- special thanks to Mike Auty for providing the info needed to fix it.
- Some racing conditions could corrupt the Pydev indexing (the effect was that the globals browser and context-insensitive code-completion would sometimes not find some tokens that were supossed to be there).
p.s.: The main update urls now only contain versions 1.4.3 and onwards (those are now using the new p2 update site structure, so, please check http://fabioz.com/pydev/download.html and http://pydev.sourceforge.net/download.html to see how to obtain some earlier version -- if you ever need it)
Friday, February 06, 2009
Pydev Nightly builds
Yeap, it's available for those that really want to keep up with the most up-to-date code...
The nightly build is available in the format of an update site.
The update site links to use are:
http://nightly.aptana.com/pydev/site.xml
http://nightly.aptana.com/pydev-pro/site.xml
Instructions on how to use it are available at:
http://nightly.aptana.com/pydev/index.html
http://nightly.aptana.com/pydev-pro/index.html
Until now, Pydev didn't have an easy way for people to get updates until a proper release was all wrapped up, so, this should be pretty handy for those of you expecting a bugfix -- or just wanting to experiment with the latest thing you're waiting for. Also, usually before committing I do run the Pydev test-suite (which is pretty big), so, it should be regularly pretty stable.
Another thing: I've been having some complains on people not being able to access fabioz.com, even with the server being up (it appears it's related to some specific providers), so, that update site can be used as a second choice if you're not currently able to access fabioz.com to get Pydev Extensions (also note that we're -- still -- in the process of moving Pydev things to Aptana, so, hopefully as soon as that's done, no one else will have any problems accessing the Pydev site).
A note for those expecting bugfixes: The Pydev builds now contain not only the current version, but also the repository id in the build (e.g.: 1.4.3.2123), so, if the svn version for a fix is known (I usually post it when I fix a bug), you can wait for the build to get to that version and then get it ( the current nightly build version can be verified at http://nightly.aptana.com/pydev/site.xml and http://nightly.aptana.com/pydev-pro/site.xml )
The nightly build is available in the format of an update site.
The update site links to use are:
http://nightly.aptana.com/pydev/site.xml
http://nightly.aptana.com/pydev-pro/site.xml
Instructions on how to use it are available at:
http://nightly.aptana.com/pydev/index.html
http://nightly.aptana.com/pydev-pro/index.html
Until now, Pydev didn't have an easy way for people to get updates until a proper release was all wrapped up, so, this should be pretty handy for those of you expecting a bugfix -- or just wanting to experiment with the latest thing you're waiting for. Also, usually before committing I do run the Pydev test-suite (which is pretty big), so, it should be regularly pretty stable.
Another thing: I've been having some complains on people not being able to access fabioz.com, even with the server being up (it appears it's related to some specific providers), so, that update site can be used as a second choice if you're not currently able to access fabioz.com to get Pydev Extensions (also note that we're -- still -- in the process of moving Pydev things to Aptana, so, hopefully as soon as that's done, no one else will have any problems accessing the Pydev site).
A note for those expecting bugfixes: The Pydev builds now contain not only the current version, but also the repository id in the build (e.g.: 1.4.3.2123), so, if the svn version for a fix is known (I usually post it when I fix a bug), you can wait for the build to get to that version and then get it ( the current nightly build version can be verified at http://nightly.aptana.com/pydev/site.xml and http://nightly.aptana.com/pydev-pro/site.xml )
Subscribe to:
Posts (Atom)