Monday, December 22, 2008
Pydev 1.4.1
Ok, after 1.4 a bug was found dealing with the licensing (for Pydev Extensions on Eclipse 3.4.1), so, this release fixes this one bug (so, it's only released again in fabioz.com, not on sourceforge).
Pydev 1.4
Yeap, Pydev 1.4 It's finally out ;-)
Major things include:
- Support for Python 3.0 and Python 2.6
- The context-based find definition was moved from Pydev Extensions to Pydev Open source (note that the context-independent part, which goes on to find a definition from all the available tokens in your workspace is still only available in Pydev Extensions)
- A major bug was fixed in the parser when multiple threads were making the parse (this became more evident in java 6 with its optimizations, so, if you found out you had warnings that sometimes appeared nonsense, it should be fixed now)
- Hovers were added for docstrings and to show variables when debugging
Aside from that, there are a bunch of other bugfixes available.
Note that there are some things that are probably still missing to say that the Python 3.0 support is a 100% finished (most notably, the coverage is still not available because the coverage.py module used still does not support it... so, it's waiting on that to properly support it) and as it's a really big change, there's probably a bunch of other things that are still missing... so, I'm waiting on the bug reports :)
Major things include:
- Support for Python 3.0 and Python 2.6
- The context-based find definition was moved from Pydev Extensions to Pydev Open source (note that the context-independent part, which goes on to find a definition from all the available tokens in your workspace is still only available in Pydev Extensions)
- A major bug was fixed in the parser when multiple threads were making the parse (this became more evident in java 6 with its optimizations, so, if you found out you had warnings that sometimes appeared nonsense, it should be fixed now)
- Hovers were added for docstrings and to show variables when debugging
Aside from that, there are a bunch of other bugfixes available.
Note that there are some things that are probably still missing to say that the Python 3.0 support is a 100% finished (most notably, the coverage is still not available because the coverage.py module used still does not support it... so, it's waiting on that to properly support it) and as it's a really big change, there's probably a bunch of other things that are still missing... so, I'm waiting on the bug reports :)
Saturday, December 20, 2008
Pydev: Python 2.6/3.0 support status
Finished topics on the support for Python 2.6 and Python 3.0:
- The grammar support is finished (a lot of work was done at this area, as the grammar changed quite a bit -- a good side-effect of that is that as I spent more time dealing with JavaCC and the AST construction, some good cleanups were done, and the parsing of all versions should be a bit faster now)
- Interpreter can be configured for those
- The project config can use the new grammars
- Code-completion is working
- Pretty printing of the new grammar seems to be complete
Things unfinished are:
- Debugger
- Unbuffered support (apparently, passing -u to the interpreter does not give unbuffered output anymore -- which is probably a regression bug), so, an acceptable workaround still needs to be found.
- Code coverage
- Code analysis must be checked with the new constructs
That's it... that support should finished pretty soon ;-)
- The grammar support is finished (a lot of work was done at this area, as the grammar changed quite a bit -- a good side-effect of that is that as I spent more time dealing with JavaCC and the AST construction, some good cleanups were done, and the parsing of all versions should be a bit faster now)
- Interpreter can be configured for those
- The project config can use the new grammars
- Code-completion is working
- Pretty printing of the new grammar seems to be complete
Things unfinished are:
- Debugger
- Unbuffered support (apparently, passing -u to the interpreter does not give unbuffered output anymore -- which is probably a regression bug), so, an acceptable workaround still needs to be found.
- Code coverage
- Code analysis must be checked with the new constructs
That's it... that support should finished pretty soon ;-)
Wednesday, November 26, 2008
Making code work in Python 2 and 3
Below are some tips for those interested in writing code that runs on Python 2 and Python 3 -- there are probably many other issues, but those were the ones I ran into while porting the code-completion code in Pydev (so, there's probably going to be a part 2 of this when I go on to port the debugger)
0. This may be one of the most important advices: breathe regularly while doing the porting... and be prepared to have uglier code waiting for you if you want to support both Python 2 and Python 3 -- if you do have a choice, don't try to support both versions of Python -- the way Python 3 is implemented, no one's supposed to do that.
1. Print can NEVER be used (use the write() method from objects or create your own print function -- with a different name and use it everywhere)
2. Catching exceptions putting the exception in a given var can NEVER be used (there's no compatible way of doing it in a way that's acceptable in both versions, so, just deal with it as you can using the traceback module)
3. socket.send needs bytearray: socket.send(bytearray(str, 'utf-8'))
4. socket.receive gets bytes (so, they need to be decoded: socket.recv(size).decode('utf-8')
5. Some imports:
try:
import StringIO
except:
import io as StringIO #Python 3.0
try:
from urllib import quote_plus, unquote_plus
except ImportError:
from urllib.parse import quote_plus, unquote_plus #Python 3.0
try:
import __builtin__
except ImportError:
import builtins as __builtin__ #Python 3.0
There are way too many others, so, the approach for getting it right is basically running the 2to3 with that import to see the new version of it and then making it work as it used to.
6. True, False assign: in some scripts, to support older python/jython versions, the following construct was used:
__builtin__.True = 1
__builtin__.False = 0
As True and False are keywords now, this assignment will give a syntax error, so, to keep it working, one must do:
setattr(__builtin__, 'True', 1) -- as this will only be executed if True is not defined, that should be ok.
7. The long representation for values is not accepted anymore, so, 2L would not be accepted in python 3.
The solution for something as long_2 = 2L may be something as:
try:
long
except NameError:
long = int
long_2 = long(2)
Note that if you want to define a number that's already higher than the int limit, that won't actually help you (in my particular case, that was used on some arithmetic, just to make sure that the number would be coerced to a long, so, that solution is ok -- note: if you were already on python 2.5, that would not be needed as the conversion int -> long is already automatic)
8. raw_input is now input and input should be written explicitly as eval(raw_input('enter value'))
So, to keep backwards compatibility, I think the best approach would be keeping on with the raw_input (and writing the "old input" explicitly, while removing the "new input" reference from the builtins)
try:
raw_input
except NameError:
import builtins
original_input = builtins.input
del builtins.input
def raw_input(*args, **kwargs):
return original_input(*args, **kwargs)
builtins.raw_input = raw_input
9. The compiler module is gone. So, to parse something, the solution seems to be using ast.parse and to compile, there's the builtins.compile (NOTE: right now, the 2to3 script doesn't seem to get this correctly)
0. This may be one of the most important advices: breathe regularly while doing the porting... and be prepared to have uglier code waiting for you if you want to support both Python 2 and Python 3 -- if you do have a choice, don't try to support both versions of Python -- the way Python 3 is implemented, no one's supposed to do that.
1. Print can NEVER be used (use the write() method from objects or create your own print function -- with a different name and use it everywhere)
2. Catching exceptions putting the exception in a given var can NEVER be used (there's no compatible way of doing it in a way that's acceptable in both versions, so, just deal with it as you can using the traceback module)
3. socket.send needs bytearray: socket.send(bytearray(str, 'utf-8'))
4. socket.receive gets bytes (so, they need to be decoded: socket.recv(size).decode('utf-8')
5. Some imports:
try:
import StringIO
except:
import io as StringIO #Python 3.0
try:
from urllib import quote_plus, unquote_plus
except ImportError:
from urllib.parse import quote_plus, unquote_plus #Python 3.0
try:
import __builtin__
except ImportError:
import builtins as __builtin__ #Python 3.0
There are way too many others, so, the approach for getting it right is basically running the 2to3 with that import to see the new version of it and then making it work as it used to.
6. True, False assign: in some scripts, to support older python/jython versions, the following construct was used:
__builtin__.True = 1
__builtin__.False = 0
As True and False are keywords now, this assignment will give a syntax error, so, to keep it working, one must do:
setattr(__builtin__, 'True', 1) -- as this will only be executed if True is not defined, that should be ok.
7. The long representation for values is not accepted anymore, so, 2L would not be accepted in python 3.
The solution for something as long_2 = 2L may be something as:
try:
long
except NameError:
long = int
long_2 = long(2)
Note that if you want to define a number that's already higher than the int limit, that won't actually help you (in my particular case, that was used on some arithmetic, just to make sure that the number would be coerced to a long, so, that solution is ok -- note: if you were already on python 2.5, that would not be needed as the conversion int -> long is already automatic)
8. raw_input is now input and input should be written explicitly as eval(raw_input('enter value'))
So, to keep backwards compatibility, I think the best approach would be keeping on with the raw_input (and writing the "old input" explicitly, while removing the "new input" reference from the builtins)
try:
raw_input
except NameError:
import builtins
original_input = builtins.input
del builtins.input
def raw_input(*args, **kwargs):
return original_input(*args, **kwargs)
builtins.raw_input = raw_input
9. The compiler module is gone. So, to parse something, the solution seems to be using ast.parse and to compile, there's the builtins.compile (NOTE: right now, the 2to3 script doesn't seem to get this correctly)
Thursday, November 13, 2008
Pydev & Python 3.0
One thing that should be added in the next release is support for Python 3.0 (aka Python 3k)
So, to cover that, I've started at the most basic part: being able to configure an interpreter with Python 3.0. And you guessed it: print does not work anymore, so, when I started checking how can support be added for Python 3.0 while still retaining compatibility with other versions (as most existent python library probably will have to do), something weird struck me: print is now something banned in the python world...
Not only won't I be able to use the print statement (because it's not available on Python 3.0), but I also won't be able to use the print function (because it's not available in older versions of Python), so, when considering the 2to3 tool, actually, what should be available is a 2toAny, where print is removed altogether in what's probably going to be the actual standard for python: instead of print 'x', the thing to have backward compatibility is writing: sys.stdout.write('x\n') -- which I think is really something sad -- probably a step backwards for python, where printing something used to be sane... unlike java things such as System.out.println()... But now not only it's very similar -- it also gets worse: you have to add the new line yourself.
So, in the 2toAny, what should be done is transforming print varA into sys.stdout.write('%s\n' % (varA,)) and print >> log, varA into log.write('%s\n', (varA,))-- much more pythonic right?!?
Well, aside from that, I wanted to say that besides that particular change, most other things seem to be consistent in taking the language forward... but that one seems just gratuitous breakage of most python code existent (it's not like print is used seldonly in the python world or anyone did actually dislike it).
I'm sorry about the tone of this post, but I must say I'm a bit disapointed with that particular change in Python at this stage... Am I missing something here? Is there another way to really have actual compatibility? -- Maybe defining my own print_ function and using it everywhere? Why did Python "fix" something that not only worked, but that everyone was happy about? (or wasn't?)
How about a python 2toAny tool, does someone know if such a thing exists? It's probably much more important than a 2to3 (as almost all libraries will need to keep supporting old versions and probably only few projects will actually migrate to 3.0 -- at least initially and until the libraries they use are able to support it).
So, to cover that, I've started at the most basic part: being able to configure an interpreter with Python 3.0. And you guessed it: print does not work anymore, so, when I started checking how can support be added for Python 3.0 while still retaining compatibility with other versions (as most existent python library probably will have to do), something weird struck me: print is now something banned in the python world...
Not only won't I be able to use the print statement (because it's not available on Python 3.0), but I also won't be able to use the print function (because it's not available in older versions of Python), so, when considering the 2to3 tool, actually, what should be available is a 2toAny, where print is removed altogether in what's probably going to be the actual standard for python: instead of print 'x', the thing to have backward compatibility is writing: sys.stdout.write('x\n') -- which I think is really something sad -- probably a step backwards for python, where printing something used to be sane... unlike java things such as System.out.println()... But now not only it's very similar -- it also gets worse: you have to add the new line yourself.
So, in the 2toAny, what should be done is transforming print varA into sys.stdout.write('%s\n' % (varA,)) and print >> log, varA into log.write('%s\n', (varA,))-- much more pythonic right?!?
Well, aside from that, I wanted to say that besides that particular change, most other things seem to be consistent in taking the language forward... but that one seems just gratuitous breakage of most python code existent (it's not like print is used seldonly in the python world or anyone did actually dislike it).
I'm sorry about the tone of this post, but I must say I'm a bit disapointed with that particular change in Python at this stage... Am I missing something here? Is there another way to really have actual compatibility? -- Maybe defining my own print_ function and using it everywhere? Why did Python "fix" something that not only worked, but that everyone was happy about? (or wasn't?)
How about a python 2toAny tool, does someone know if such a thing exists? It's probably much more important than a 2to3 (as almost all libraries will need to keep supporting old versions and probably only few projects will actually migrate to 3.0 -- at least initially and until the libraries they use are able to support it).
Monday, October 27, 2008
Pydev 1.3.23 and 1.3.24
Yes, that's right, 2 new releases were done.
Basically, 1.3.24 fixes a major bug in Pydev, related to the debugger (it was halting on some cases and Jython was not working because a dependency on itertools was added).
I think that the major highlight in 1.3.23 is the possibility to debug multiple processes at the same time in the remote debugger (yes, it only supported one at a time), so, if you're spawning multiple processes, that should help a bit :)
Another fix was an incompatibility with the update manager of Eclipse 3.4... It should be backward compatible, but apparently, because the Pydev update site contained some really old versions, after installing Pydev, no other plugins could be installed! -- something related to getting the dependencies wrong in the new Eclipse update manager.
Aside from that, this release had lots of minor bugs fixed, so, upgrading is highly recommended!
Basically, 1.3.24 fixes a major bug in Pydev, related to the debugger (it was halting on some cases and Jython was not working because a dependency on itertools was added).
I think that the major highlight in 1.3.23 is the possibility to debug multiple processes at the same time in the remote debugger (yes, it only supported one at a time), so, if you're spawning multiple processes, that should help a bit :)
Another fix was an incompatibility with the update manager of Eclipse 3.4... It should be backward compatible, but apparently, because the Pydev update site contained some really old versions, after installing Pydev, no other plugins could be installed! -- something related to getting the dependencies wrong in the new Eclipse update manager.
Aside from that, this release had lots of minor bugs fixed, so, upgrading is highly recommended!
Thursday, October 02, 2008
Pydev Subversion @ Aptana
I think it's already widely publicized that Pydev is now a part of Aptana, and the last release is starting to mark that integration: the source is now available at the subversion hosted by Aptana.
The basic info to get it is:
Enjoy!
The basic info to get it is:
- repo: https://source.aptana.com/pydev_repo/
- user: anonymous
- password: aptana
Enjoy!
Pydev 1.3.22
This Pydev release was focused mostly on bug-fixes, with many minor things being done. It fixed that nasty bug related to handling __all__ incorrectly together with a bunch of others.
I think that the only feature added that's worth mentioning is a mode where you can have Pydev analyze only your current editor (this enables builds to happen faster, as less work is needed when building the project, so, it can increase your performance a bit at the cost of having the markers added only to open editors). To activate it, go to window > preferences > Pydev > Builders and check 'Only analyze open editors'.
Note that doing so will not remove existing markers, and will be available for new analysis (the markers may be removed by right-clicking a folder and choosing "Pydev > Remove error markers"). Also, this feature is still in beta, so, please report any bugs found when using it.
I think that the only feature added that's worth mentioning is a mode where you can have Pydev analyze only your current editor (this enables builds to happen faster, as less work is needed when building the project, so, it can increase your performance a bit at the cost of having the markers added only to open editors). To activate it, go to window > preferences > Pydev > Builders and check 'Only analyze open editors'.
Note that doing so will not remove existing markers, and will be available for new analysis (the markers may be removed by right-clicking a folder and choosing "Pydev > Remove error markers"). Also, this feature is still in beta, so, please report any bugs found when using it.
Friday, August 15, 2008
Pydev 1.3.19
I guess 1.3.19 is already old news by now, but I decided to comment a bit on it, especially on a new support direction that'll start to be adopted in the Pydev development (and I'm still unsure how well that'll work):
Pydev will now try to support Eclipse 3.2, 3.3 and 3.4 (all with the same binaries) -- actually, there are still some known bugs on Eclipse 3.4, but those should be solved for the next release.
Now, while trying to support all those versions with the same binary release, there are some hacks that don't really seem right for me (but I couldn't see another way out for it):
1. Catching Exceptions such as ClassNotFoundException / LinkageError and related and falling back to another version (usually deprecated) of some API
2. When adapting to some interface, instead of having adapter == IElementContentProvider.class, do things as if(adapter.toString().endsWith("org.eclipse.debug.internal.ui.viewers.model.provisional.IElementContentProvider"))
-- because that class still does not exist on a previous version.
3. Using reflection (yes, in languages as Python this would be the default, but Java's not really made for this)
BTW: This wouldn't be possible without using Mercurial locally -- right now I have 4 workspaces, all synched through mercurial -- one for each version of Eclipse and one that's the link to the outside world -- and each of those has some changes to make it compile in that version -- yes, the final code runs in all versions, but it'll only actually compile in 3.3 for now -- the others have classes not defined (which are those cases with ClassNotFoundException), so, the code must be tested later on those versions with the final release.
Pydev will now try to support Eclipse 3.2, 3.3 and 3.4 (all with the same binaries) -- actually, there are still some known bugs on Eclipse 3.4, but those should be solved for the next release.
Now, while trying to support all those versions with the same binary release, there are some hacks that don't really seem right for me (but I couldn't see another way out for it):
1. Catching Exceptions such as ClassNotFoundException / LinkageError and related and falling back to another version (usually deprecated) of some API
2. When adapting to some interface, instead of having adapter == IElementContentProvider.class, do things as if(adapter.toString().endsWith("org.eclipse.debug.internal.ui.viewers.model.provisional.IElementContentProvider"))
-- because that class still does not exist on a previous version.
3. Using reflection (yes, in languages as Python this would be the default, but Java's not really made for this)
BTW: This wouldn't be possible without using Mercurial locally -- right now I have 4 workspaces, all synched through mercurial -- one for each version of Eclipse and one that's the link to the outside world -- and each of those has some changes to make it compile in that version -- yes, the final code runs in all versions, but it'll only actually compile in 3.3 for now -- the others have classes not defined (which are those cases with ClassNotFoundException), so, the code must be tested later on those versions with the final release.
Saturday, June 21, 2008
Configuring tvtk / forced builtins
Recently, there was a report on problems while configuring tvtk (it's a pythonic API over vtk).
Having the import:
from enthought.tvtk.api import tvtk
did not bring any completions while later using 'tvtk.' on that module.
This happens because the code in tvtk does some black-magic to load the classes in that module... there's a module called enthought.tvtk.tvtk that has a class tvtk that's actually what's imported in "from enthought.tvtk.api import tvtk", so, the solution is adding "enthought.tvtk.tvtk.tvtk" to the "forced builtins", so that pydev can understand that it should be dynamically analyzed.
More info on the "forced builtins" can be found at: http://www.fabioz.com/pydev/manual_101_interpreter.html... and yes: finding out what should be dynamically analyzed and what shouldn't can be very tricky some times...
Having the import:
from enthought.tvtk.api import tvtk
did not bring any completions while later using 'tvtk.' on that module.
This happens because the code in tvtk does some black-magic to load the classes in that module... there's a module called enthought.tvtk.tvtk that has a class tvtk that's actually what's imported in "from enthought.tvtk.api import tvtk", so, the solution is adding "enthought.tvtk.tvtk.tvtk" to the "forced builtins", so that pydev can understand that it should be dynamically analyzed.
More info on the "forced builtins" can be found at: http://www.fabioz.com/pydev/manual_101_interpreter.html... and yes: finding out what should be dynamically analyzed and what shouldn't can be very tricky some times...
Pydev 1.3.18
The major feature on this release was the (organize imports feature, along with some bug-fixes...
There's even one bug-fix I think it's worth talking about:
Pydev was using Runtime.exec(cmd) to execute the shell/gather interpreter info, and that wasn't very portable among platforms, resulting in problems when paths contained spaces. Now, in 1.3.18, the Runtime.exec(cmd[]) (using an array to pass parameters) is being used, and that should fix the 'spaces on install' problem -- actually, there was a partial fix on 1.3.17 for that, but some places still used the wrong API... now, on 1.3.18, all places should be using the new API.
There's even one bug-fix I think it's worth talking about:
Pydev was using Runtime.exec(cmd) to execute the shell/gather interpreter info, and that wasn't very portable among platforms, resulting in problems when paths contained spaces. Now, in 1.3.18, the Runtime.exec(cmd[]) (using an array to pass parameters) is being used, and that should fix the 'spaces on install' problem -- actually, there was a partial fix on 1.3.17 for that, but some places still used the wrong API... now, on 1.3.18, all places should be using the new API.
Sunday, June 15, 2008
Organizing imports
One of the features I can't live without in Pydev Extensions is the auto-import. Basically, when you want to get a class/method into the namespace, just start writing it, request a completion and the class will be completed and an import for it will be added at the top-level part of the file (it's similar to what JDT does).
So, after using this feature lots of times, one thing that started happening is that some modules would end with imports such as:
from mod1 import Class1
from mod1 import Class2
from mod1 import Class3
Now, on version 1.3.18, the auto-import will group imports when requested, so, after making those requests it'd become:
from mod1 import Class1, Class2, Class3
But that won't group existing imports... For that, the organize imports action was improved -- until now, Ctrl+Shift+O would only sort the imports alphabetically, but from now on it'll also group them, and if Pydev Extensions is available, for each undefined token found in the code analysis, an option to add an import to one of the available definitions of the token will be presented -- which is especially useful when doing refactorings and cutting/copying code from one place to another.
So, after using this feature lots of times, one thing that started happening is that some modules would end with imports such as:
from mod1 import Class1
from mod1 import Class2
from mod1 import Class3
Now, on version 1.3.18, the auto-import will group imports when requested, so, after making those requests it'd become:
from mod1 import Class1, Class2, Class3
But that won't group existing imports... For that, the organize imports action was improved -- until now, Ctrl+Shift+O would only sort the imports alphabetically, but from now on it'll also group them, and if Pydev Extensions is available, for each undefined token found in the code analysis, an option to add an import to one of the available definitions of the token will be presented -- which is especially useful when doing refactorings and cutting/copying code from one place to another.
Monday, May 12, 2008
Pydev 1.3.17 released
Ok, this release is just to iron out some high-priority bugs that have been introduced in the last version.
Most notably, a bug where the pydev package explorer was not working correctly when the project root was configured as a source folder and some fixes in the debugger step return / step over (because of improvements to run those with untraced frames).
Yeap, seems that the saying "Given enough eyeballs, all bugs are shallow" is really true -- I had already been running 1.3.16 for some days here and after the release it took just a couple of hours for those bugs to be reported.
Enjoy!
Most notably, a bug where the pydev package explorer was not working correctly when the project root was configured as a source folder and some fixes in the debugger step return / step over (because of improvements to run those with untraced frames).
Yeap, seems that the saying "Given enough eyeballs, all bugs are shallow" is really true -- I had already been running 1.3.16 for some days here and after the release it took just a couple of hours for those bugs to be reported.
Enjoy!
Thursday, May 08, 2008
Bug in pydev package explorer (1.3.16)
Ok, a serious bug was found in the pydev package explorer... in version 1.3.16, when a project has the project root in the pythonpath, its children won't appear. For users that have a source folder within the project, this doesn't seem to happen.
This problem has just been fixed and a new release should be out pretty soon (in 1.3.17)
This problem has just been fixed and a new release should be out pretty soon (in 1.3.17)
Wednesday, May 07, 2008
Pydev 1.3.16
Yeap, it's just been released. Most of the work on this release was on bug-fixes, so, it should be safe to upgrade without major concerns.
The launching facility had some changes, mostly regarding the Ctrl+F11 when it's set to launch the current editor (it just didn't work the way it was supposed to), but that's only valid for new launch configurations (so, if you do want to use it instead of having it launch the previously launched app, existing launches should be deleted -- just note that you should delete only a few launches at a time (around 10-15) -- for some reason eclipse takes a lot of time to delete lots of launches at once (I tried doing it here and it halted for about 5 minutes until I decided to kill it and delete in small steps).
The launching facility had some changes, mostly regarding the Ctrl+F11 when it's set to launch the current editor (it just didn't work the way it was supposed to), but that's only valid for new launch configurations (so, if you do want to use it instead of having it launch the previously launched app, existing launches should be deleted -- just note that you should delete only a few launches at a time (around 10-15) -- for some reason eclipse takes a lot of time to delete lots of launches at once (I tried doing it here and it halted for about 5 minutes until I decided to kill it and delete in small steps).
Wednesday, April 09, 2008
Pydev 1.3.15 released (new interactive console)
The major highlight in this release is the new interactive console. Details on it can be found at: http://pydev.sourceforge.net/console.html.
Pydev extensions also has some pluses regarding the interactive console (see: http://fabioz.com/pydev)
I'll do a video showing it in more details shortly (probably over the weekend).
This release also has some other nice things, such as having the regular console accepting unicode chars that are not in the ascii range (it uses the sitecustomize.py feature from python to set the correct encoding to be used in the shell) and other bugfixes.
Get it while it's fresh :)
Pydev extensions also has some pluses regarding the interactive console (see: http://fabioz.com/pydev)
I'll do a video showing it in more details shortly (probably over the weekend).
This release also has some other nice things, such as having the regular console accepting unicode chars that are not in the ascii range (it uses the sitecustomize.py feature from python to set the correct encoding to be used in the shell) and other bugfixes.
Get it while it's fresh :)
Thursday, March 20, 2008
Interactive console in Pydev
Last week I started working on an interactive console and bundling it into pydev. Hopefully it'll be finished soon (probably within the next 1-2 weeks).
The main structure is already in place... code-completion, context info on hover, history (up/down arrow), colors for stdout/err/prompt/user input, auto-indent, python/jython support, it's all there already... (but I want to add an extra layer of polishment before releasing it).
Under the hood, it's almost like a whole new editor covering the console, while communicating through sockets with a python shell (but it feels a lot like a regular editor, in the sense that the same options that are used in the editor for indent, code completion, etc. are also available from the console -- with the exception that you can actually only edit the active line).
Also, this time I was able to get some code that had an initial draft of a console from dltk (which was pretty nice), but I had to do some (not so minor) changes to be able to add it to pydev, although the basic structure still has the same idea (the result can be seen at org.python.pydev/src_dltk_console).
Well, to sum it up... this is a long-awaited feature, and the plans are for surpassing all expectations on that :)
Friday, March 07, 2008
Pydev 1.3.14
Pydev 1.3.14 is out!
It's mostly a bugfix release with some of niceties:
* In pydev extensions, configuring the remote debugger in a different machine is now easier: mostly setting the constant in pydevd_file_utils.py (in the remote machine being debugged -- see the comments on the file).
* In pydev, having a speed up from using psyco was really nice!
* The improvements in the outline view (by Laurent Dore) and having a more reliable Go to Next/Previous method (Ctrl+Shift+Up/Down) are also pretty nice :)
It's mostly a bugfix release with some of niceties:
* In pydev extensions, configuring the remote debugger in a different machine is now easier: mostly setting the constant in pydevd_file_utils.py (in the remote machine being debugged -- see the comments on the file).
* In pydev, having a speed up from using psyco was really nice!
* The improvements in the outline view (by Laurent Dore) and having a more reliable Go to Next/Previous method (Ctrl+Shift+Up/Down) are also pretty nice :)
Saturday, February 16, 2008
Pydev debugger and psyco speedups (target: 1.3.14)
The pydev debugger can now make use of psyco if it's available in the environment (it's actually highly recommended... in tests a 50% improvement in the debugger speed was observed... mainly, all the tracing goes through 2 functions, so, speeding it was pretty straightforward -- after discovering how to integrate it).
Now, this relation is kind of strange: the pydev debugger can make use of psyco, but the debugged code can't (because if it gets compiled, pydev can't trace it).
Until now, a Null() object was added as a psyco module in sys.modules so that clients didn't worry about it, but that had some nasty side effects, so, this version will also change this integration so that clients that rely on psyco don't have to change code: a stub module will replace the psyco module in sys.modules and will mimic it better -- and it'll be installed only after pydev finishes using it to get its optimizations.
Now, this relation is kind of strange: the pydev debugger can make use of psyco, but the debugged code can't (because if it gets compiled, pydev can't trace it).
Until now, a Null() object was added as a psyco module in sys.modules so that clients didn't worry about it, but that had some nasty side effects, so, this version will also change this integration so that clients that rely on psyco don't have to change code: a stub module will replace the psyco module in sys.modules and will mimic it better -- and it'll be installed only after pydev finishes using it to get its optimizations.
Saturday, February 02, 2008
Pydev 1.3.12: bug in outline
The release has been out for some days now, but there's one bug in that release I think is worth mentioning: the outline is not behaving as it should: the actions are not there and it doesn't link with the editor.
This is already fixed in the cvs and will be released next week in a bugfix release...
Nice tip: Always remember to check if a weak-reference is still alive before trying to access it -- this specific portion of the code hasn't been changed in a while, but after changing other things this bug started to show...
Ain't it amazing how software works? I wonder if this kind of thing will be a thing of the past at some point in the future...
Testing for common situations and expanding those tests as bugs are found seems to be the best alternative (currently) -- this is a good feature on dynamic languages: you're never left thinking that if it compiles it works... that's never true anyways -- but it would be nice if there was a better way to ensure things work :)
I guess code-analysis helps there, but it is surely not foolproof... and the number of false positives must also be taken into account: that's one of the reasons why 'self' access check has not been added to pydev extensions: the number of false positives in some sample programs was showing that it was too common to access attributes that the code-analysis couldn't get.
If you think in the dynamic world, code-analysis is also more limited (as there's less info to work on)... The pydev extensions code-analysis helps while you're programming in the same way that a compiler would help you catch stupid errors... and having it does not ensure the program will work anyways (although I think it makes life much more pleasant!... I can't even imagine myself programming in python without having those errors gotten by the code-analysis... the code-run cycle is much slower without it, mostly because the run part ends up acting as the code-analysis -- which should have happened in the code part).
And you know... after you know the reason for the bug it is usually easy fixing it. The hard part is always discovering the bug (which is what we try to tackle with tests, code-analysis, code-reviews... ... ... )!
This is already fixed in the cvs and will be released next week in a bugfix release...
Nice tip: Always remember to check if a weak-reference is still alive before trying to access it -- this specific portion of the code hasn't been changed in a while, but after changing other things this bug started to show...
Ain't it amazing how software works? I wonder if this kind of thing will be a thing of the past at some point in the future...
Testing for common situations and expanding those tests as bugs are found seems to be the best alternative (currently) -- this is a good feature on dynamic languages: you're never left thinking that if it compiles it works... that's never true anyways -- but it would be nice if there was a better way to ensure things work :)
I guess code-analysis helps there, but it is surely not foolproof... and the number of false positives must also be taken into account: that's one of the reasons why 'self' access check has not been added to pydev extensions: the number of false positives in some sample programs was showing that it was too common to access attributes that the code-analysis couldn't get.
If you think in the dynamic world, code-analysis is also more limited (as there's less info to work on)... The pydev extensions code-analysis helps while you're programming in the same way that a compiler would help you catch stupid errors... and having it does not ensure the program will work anyways (although I think it makes life much more pleasant!... I can't even imagine myself programming in python without having those errors gotten by the code-analysis... the code-run cycle is much slower without it, mostly because the run part ends up acting as the code-analysis -- which should have happened in the code part).
And you know... after you know the reason for the bug it is usually easy fixing it. The hard part is always discovering the bug (which is what we try to tackle with tests, code-analysis, code-reviews... ... ... )!
Tuesday, January 29, 2008
Pydev Release 1.3.12 and bug-reporting changes
Pydev 1.3.12 is out!
The major change is the speed-up in operations that require type-inference (e.g.: code-completion, code-analysis, etc.) and some bug-fixing!
Aside from that, from now on, bug reports will need a valid user registered at sourceforge... the main rationale for this is that most of the reports that come from anonymous users are usually not properly fixed because lots of times the 1st report misses the basic info to reproduce the bug, and if reporters can't be reached those end up being pretty much useless...
The major change is the speed-up in operations that require type-inference (e.g.: code-completion, code-analysis, etc.) and some bug-fixing!
Aside from that, from now on, bug reports will need a valid user registered at sourceforge... the main rationale for this is that most of the reports that come from anonymous users are usually not properly fixed because lots of times the 1st report misses the basic info to reproduce the bug, and if reporters can't be reached those end up being pretty much useless...
Friday, January 25, 2008
Eclipse Jobs API and Pydev Optimizations
1st thing to learn: Don't use UIJobs unless it's imperative that you have access to the UI... From 1.3.9 to 1.3.10 I was doing some refactorings to use the Eclipse Jobs API instead of raw Threads...
After taking a look at the core options (Job, UIJob and WorkbenchJob), the WorkbenchJob was chosen, as things shouldn't run if the workbench was shutting down (the docs say that it'll do checks to see if the Workbench is running, before actually running the Job -- as the Job in case: code-analysis is not suitable to run at shutdown, it seemed a nice choice)
But the thing is that the workbench job ends up running in the UI Thread -- and thus gives a little halt to the user every time the job runs... (and only after going after it in a bug report I found out about it).
On the good side, I took some more time to profile the code-analysis / code-completion (mainly when interacting with large files) and was able to make some nice improvements that will surely be noticeable... even on smaller cases:
- a cache that works for the duration of a type-analysis request to hold non-direct tokens available at a module was added
- a cache to hold scope information within a module was also created...
In my tests it gave a 40x improvement on a type-analysis request, as many traverses of the ASTs are not needed anymore with those caches (those will have a higher impact when a given module is imported a lot from different places).
All in all, optimizing Pydev is becoming more and more challenging :)
p.s.: Those improvements will be available on Pydev 1.3.12 (which hopefully will be released in the beginning of the next week).
After taking a look at the core options (Job, UIJob and WorkbenchJob), the WorkbenchJob was chosen, as things shouldn't run if the workbench was shutting down (the docs say that it'll do checks to see if the Workbench is running, before actually running the Job -- as the Job in case: code-analysis is not suitable to run at shutdown, it seemed a nice choice)
But the thing is that the workbench job ends up running in the UI Thread -- and thus gives a little halt to the user every time the job runs... (and only after going after it in a bug report I found out about it).
On the good side, I took some more time to profile the code-analysis / code-completion (mainly when interacting with large files) and was able to make some nice improvements that will surely be noticeable... even on smaller cases:
- a cache that works for the duration of a type-analysis request to hold non-direct tokens available at a module was added
- a cache to hold scope information within a module was also created...
In my tests it gave a 40x improvement on a type-analysis request, as many traverses of the ASTs are not needed anymore with those caches (those will have a higher impact when a given module is imported a lot from different places).
All in all, optimizing Pydev is becoming more and more challenging :)
p.s.: Those improvements will be available on Pydev 1.3.12 (which hopefully will be released in the beginning of the next week).
Tuesday, January 15, 2008
Pydev release: 1.3.11
This release fixed some problems on the 3.3 integration (such as the undo/redo limit not being respected and the new 'insert spaces for tabs' in the general preferences conflicting with the pydev options).
Some niceties, such as being able to define custom filters in the pydev package explorer, not collapsing things in the outline unless really needed and having spell checking (which still depends on JDT -- because the Eclipse platform will only have that when 3.4 is released) are also there...
Now, the actual highlights were on the jython side this time: Java projects can now be referenced and have code-completion and go-to-definition working (actually, the go-to-definition is only available in pydev extensions) and the jython debugger should be finally working... There's even a little story about it:
The debugger used to work strangely in jython -- almost randomly... After lots of investigation, it turned up that pydev had a thread that was started with a delay to keep checking events in the debugger. That thread tried to run untraced, and changed sys.settrace(None) -- at that time, all stopped working (easy to say that now, but I had to go, get jython, compile it and run the debugger in a 'mixed mode' with the JDT/pydev debugger to find out about that).
In the end, changing sys.settrace in jython didn't only change the current thread tracing, but for the whole system... Paul Jean had already submitted a patch (which was applied to the trunk yesterday) so that that threads can run untraced... meanwhile, the pydev debugger is running with all threads traced on jython (checking for the actual version, so, as soon as a new jython release comes out, it should already be working).
Enjoy ;-)
Some niceties, such as being able to define custom filters in the pydev package explorer, not collapsing things in the outline unless really needed and having spell checking (which still depends on JDT -- because the Eclipse platform will only have that when 3.4 is released) are also there...
Now, the actual highlights were on the jython side this time: Java projects can now be referenced and have code-completion and go-to-definition working (actually, the go-to-definition is only available in pydev extensions) and the jython debugger should be finally working... There's even a little story about it:
The debugger used to work strangely in jython -- almost randomly... After lots of investigation, it turned up that pydev had a thread that was started with a delay to keep checking events in the debugger. That thread tried to run untraced, and changed sys.settrace(None) -- at that time, all stopped working (easy to say that now, but I had to go, get jython, compile it and run the debugger in a 'mixed mode' with the JDT/pydev debugger to find out about that).
In the end, changing sys.settrace in jython didn't only change the current thread tracing, but for the whole system... Paul Jean had already submitted a patch (which was applied to the trunk yesterday) so that that threads can run untraced... meanwhile, the pydev debugger is running with all threads traced on jython (checking for the actual version, so, as soon as a new jython release comes out, it should already be working).
Enjoy ;-)
Sunday, January 06, 2008
Integration with java projects
The only thing keeping me from doing a new release is having the integration with JDT finished.
Pydev is already able (in the cvs version) to deal with java projects in a basic level (code-completion is there), but some things as find definition (among others) are missing.
Currently when a request for getting the completions is issued, a wrapper to manage referenced java projects is created (and that reference will be kept until the completion finishes) -- that's done in ProjectModulesManager#addModuleManagers
That 'modules manager' for java (namely: JavaProjectModulesManager) is able to create the java modules for classes it's able to resolve in that project: JavaProjectModulesManager#getModuleInDirectManager (modules that come from the system configuration are discarded -- that's handled in the jython system configuration already)
In that structure, each java class is internally mapping to a jython module and the completions are gathered from those modules -- those completions are transformed into ITokens to fit nicely the internal structure:
- AbstractJavaClassModule: has a collector for getting the completions and transforming them to ITokens;
- JavaClassModuleInProject: uses that collector for getting tokens at a java project;
- JavaZipClassModule: uses that collector for getting tokens in the system jars.
Yesterday I spent most of the day setting up a unit-test so that I could test those things (JavaClassModuleTestWorkbench), but now that it's done, I hope things will flow better in that integration (those JDT APIs are not really done for use outside of the JDT structure, so I've spent lot's of time to make it work -- and I'll probably spend some more time to make it work 'right'...)
Some of the doubts I had in the process:
JDT code-completion for external jar in non java project: Implemented at: JavaZipClassModule
API to find out about available JDT packages/classes in a project: Implemented at: JavaClassModuleInProject
Pydev is already able (in the cvs version) to deal with java projects in a basic level (code-completion is there), but some things as find definition (among others) are missing.
Currently when a request for getting the completions is issued, a wrapper to manage referenced java projects is created (and that reference will be kept until the completion finishes) -- that's done in ProjectModulesManager#addModuleManagers
That 'modules manager' for java (namely: JavaProjectModulesManager) is able to create the java modules for classes it's able to resolve in that project: JavaProjectModulesManager#getModuleInDirectManager (modules that come from the system configuration are discarded -- that's handled in the jython system configuration already)
In that structure, each java class is internally mapping to a jython module and the completions are gathered from those modules -- those completions are transformed into ITokens to fit nicely the internal structure:
- AbstractJavaClassModule: has a collector for getting the completions and transforming them to ITokens;
- JavaClassModuleInProject: uses that collector for getting tokens at a java project;
- JavaZipClassModule: uses that collector for getting tokens in the system jars.
Yesterday I spent most of the day setting up a unit-test so that I could test those things (JavaClassModuleTestWorkbench), but now that it's done, I hope things will flow better in that integration (those JDT APIs are not really done for use outside of the JDT structure, so I've spent lot's of time to make it work -- and I'll probably spend some more time to make it work 'right'...)
Some of the doubts I had in the process:
JDT code-completion for external jar in non java project: Implemented at: JavaZipClassModule
API to find out about available JDT packages/classes in a project: Implemented at: JavaClassModuleInProject
Subscribe to:
Posts (Atom)