Package Explorer status for Pydev

Well, the cvs implementation already had some nice improvements when compared to the current version (such as working with a working set), but there is one issue that is worrying me.

Basically, the current Eclipse CNF (Common Navigator Framework) has a critical bug for me, meaning that it has a little delay when synchronizing actions with the selected resource (the bug is reported at the eclipse bugzilla ), and it can be nasty... got me to delete a whole source folder instead of a single file, so, I guess I'll have to replace all the actions for the resource actions before actually considering it mature... Still, it already is a good way for browsing your project (although it still does not have the 'flat' mode implemented).

What's going on in pydev

Ok, almost 3 weeks without posting anything...

I'm still working in the package explorer implementation for pydev. I think most current bugs are solved, and what's still missing before a release is handling working sets correctly (which I hope won't take too much).

After taking a look at the way I should handle it, it appears that my ITreeContentProvider implementation (hierarchy below) must be able to handle an IWorkingSet:

BaseWorkbenchContentProvider (ITreeContentProvider)
--> PythonBaseModelProvider (IResourceChangeListener, IPythonNatureListener)
----> PythonModelProvider (IPipelinedTreeContentProvider)

So, basically the inputChanged(Viewer viewer, Object oldInput, Object newInput) method from the PythonBaseModelProvider must be extended so that it works accordingly if the newInput is an IWorkingSet. I hope it won't prove too difficult (but I thought that creating the IPipelinedTreeContentProvider -- which is basically replacing objects that are from the resource domain for objects that are from the pydev domain -- would be very easy, but it ended up not being that straightforward (anyway it seems that it is already working correctly).

Traveling

Just wanted to say that I'll be traveling from November 15th-30th... So, if I don't answer your e-mails in a timely manner during that period, please be patient, as I'll only have limited internet access...

Next steps

Well, one natural way to go would be improving the package explorer (yes, it's still in beta), so, I hope to improve it for the next version...

Note: I'm currently getting used to put things I need to do in pydev in FreeMind I think it's a pretty nice tool for planning stuff... Many of the notes I have could probably become feature requests, but I find it nicer to manage those issues through FreeMind, as it is much less bureaucratic and I have a much better view on what needs to be done -- although I don't think it is suited for bug management... for which a bug-reporting tool such as the one sourceforge provides is probably better.

So, in my notes, the things that need to be achieved in the package explorer are:
- Creating a filter for .pyc files
- Working with a working set
- Seeing the modules in a flat layout

After getting those implemented and testing it more, then I'd stop calling it beta ;-)

Pydev 1.2.5 released

This release has been pretty nice... It features an early release of a Pydev Package Explorer, using the Eclipse CNF (Common Navigator Framework) -- it still has some rough edges, but I hope to have that fixed for the next release...

Also, some contributions were pretty nice... Gergely Kis provided a patch for creating a '.pydevproject' file with the configuration for the project pythonpath and another for having a 'default interpreter' option when running some file and Gregory Golberg provided a patch for a 'quick-display' selected variable when debugging (Ctrl+Shift+D)

And as usual... there were also some bug-fixes.

Issues doing the Package Explorer for Pydev

Yeap, sometimes developing in Eclipse can become hard... especially when things don't go your way ;-)

The major problem for doing the Package Explorer in Pydev is that the CNF (Common Navigator Framework) that is provided in Eclipse has some deficiencies when it commes to providing the default actions that are provided to the default resources (IFile, IFolder) for objects that are not really implementations of those interfaces, but should be able to adapt to that interface.

The first stop in the way was that you have to provide a factory to the adapt protocol, whereas it is common in Eclipse to just make your object IAdaptable and have it working... after having that solved, it appears that the default actions only work on actual implementations, and not objects that adapt to that interface... maybe a sign that the CNF is still not completely ready (still has lots of rough edges)...

Anyway, I hope to have a first working version in the next release -- there are already some things that may deserve a release, such as some significant changes in the debugger and some niceties provided by Gergely Kis to save the project pythonpath in a file to be added to the repository and having a 'default interpreter' in the combo for selecting which interpreter to use for a run.

Package Explorer for Pydev

Ok, I've been thinkering about the next thing to do in pydev, and after taking a look at several options, creating a package explorer (instead of just using the regular navigator) seemed like a nice addition -- that's actually an old request but only when Eclipse 3.2 was added, a 'default' way of doing it was provided, so, here we go...

After searching the net for some time, the best resources to help in this implementation were:

• http://scribbledideas.blogspot.com/2006/05/building-common-navigator-based-viewer.html
  


• http://scribbledideas.blogspot.com/2006/06/what-does-common-navigator-framework.html
  


• http://scribbledideas.blogspot.com/2006/05/building-common-navigator-based-viewer_22.html
  


• http://wiki.eclipse.org/index.php/Common_Navigator_Framework

After passing the first and second tutorial in the blog, it became clear how to actually add content to make the outline of the python files in the custom navigator... I've already checked in a version that has a basic outline for python classes, but there are still some points to play with, such as making the sort of the tokens from files and making the flat/hierarchical view.

The working sets will not be available right now, as this 'common navigator framework' still does not support it (but it appears that Eclipse 3.2.1 will support it).

Configuring pydev to work with django

Update: Pydev now has a new Django integration. See: http://pydev.org/manual_adv_django.html



Ok, some time ago I've covered how to configure pydev to work with Turbogears, and since them, I've been having some requests to do the same for Django, so... here we go ;-)

As with Turbogears, I still don't have any previous experience working with Django, so, it may be good to see how easy it is to get started with it for others!

First step: get Django. Obvious place (after searching for it): http://www.djangoproject.com/

I started by reading the overview, and it seems to be straightforward... now on to something real: downloading it... it says it only needs python, but I didn't find information on which is the preferred version. I googled for it and it appears that it is still not working well with python 2.5, so, I'll go with python 2.4.

Second step: install

[C:\bin\Python24]python e:\Install\Dev.Python\Django-0.95\setup.py install
>> The required version of setuptools (>=0.6c1) is not available, and can't be installed while this script is running. Please install a more recent version first.

The tutorial says that it will attempt to grab setuptools if setuptools is not there, but unfortunatelly it seems it does not upgrade if a version is already there.

So, after googling a little, I found http://peak.telecommunity.com/DevCenter/EasyInstall#installing-easy-install and discovered that I had to download an ez_setup.py script and execute it. It appears to work without any problems, so, on to trying to install django again...

[C:\bin\Python24]python e:\Install\Dev.Python\Django-0.95\setup.py install
still gives me problems, because it requires me to be in the same folder where setup.py is...

[E:\Install\Dev.Python\Django-0.95]c:\bin\Python24\python.exe setup.py install
seems to solve it... smoke test: python -c "import django" seems to work ok

It also requires a database (as I'm just testing, pysqlite seems a good option -- I'm using version 2.3.0)

Now, on to re-configure the pydev interpreter to add those new paths... I'm kind of lazy, so, usually I just remove the interpreter and add it again, instead of figuring out just the new paths (I also thinks it's safer, as it gets all the folders it finds in the pythonpath automatically). More info on that at http://www.fabioz.com/pydev/manual_101_interpreter.html

Now, on to the actually doing something

To create a base project it provides a script, so let's do it:

django-admin.py startproject mysite

It created a 'mysite' folder with some files... still in the command line (inside the mysite folder), it says to start django with:

python manage.py runserver

And voila... it's running. Now, on to do that from inside Eclipse.

Eclipse requires your code to be under a project, so, the 'new pydev project' wizard should be used, pointing the project contents to the 'mysite' folder or to a parent folder. As there is usually much more than just code in a project -- like folders for python, c++, docs... and as in this example we surely won't have a c++ folder, the initial project structure became:
/mysite
    /docs
    /src <-- this is the folder that should be added to the project pythonpath (in the project properties)
        /mysite <-- I moved the initial files to this folder (so that we can make "import mysite" as the src is added to the pythonpath)

-- for more details on creating a project and setting the project pythonpath, check: http://www.fabioz.com/pydev/manual_101_project_conf.html

Now, on to starting the server from within pydev:
If you just run the manage.py file (just open it and press F9), it will show you a message with its usage, so, run it once and then go to the menu: Run > Run... and select the 'mysite manage.py' run and put in the program arguments runserver.

Now, running it like that leaves us without any prompt inside of the Eclipse console. Taking a closer look in the django code, it appears to have an autoreloader structure (I guess that's something common to web-servers), so, as it is actually run in another process, we won't have any output. To fix this in the testing environment (after killing the process inside of Eclipse and the other process that is still running in the computer), we can just add another argument, so, now as arguments we have runserver --noreload

Note that the Eclipse console currently does not have a Ctrl+Break, so, you have to actually kill it in the red button in the console, and to re-run the last run, just use Ctrl+F11.

Now, on to set the sqlite parameters and create the actual database -- according to the tutorial, just set the engine and file to be used for sqlite and the on to:

python manage.py syncdb

-- I usually do that stuff on a separate prompt, as it is nice to run things inside of Eclipse, but having to change the argument for each different run is not all that nice (so, an open shell is always welcome)

and created a superuser... if you configured the auto-refresh for Eclipse as in http://www.fabioz.com/pydev/manual_101_eclipse.html, a new file should be appearing in the database file for sqlite -- if you pointed it to some file inside of your project (if you still haven't the auto-refresh configure, I really advise doing it, as it can be a pain having to F5 for it to refresh the folders when some external change is done to the filesystem)

Now, keeping on with the tutorial:

python manage.py startapp polls

and copied the example from the homepage. Defining the model seems straightforward, and code-completion / code-analysis should be already working ;-)

Making the syncdb again updates my sql with the new structure (pretty nice).

Now, adding the admin interface seems pretty straightforward, and without the auto-reload, the regular debugger works pretty well -- test it by putting a breakpoint after creating the __str__ function for the Poll class in the example and pressing F11 (used to debug the last run). If you want to keep the auto-reloader and still debug, you'll have to use the remote debugger that comes with the extensions to debug, as it does not rely on having the process started from Eclipse. Check http://www.fabioz.com/pydev/manual_adv_remote_debugger.html for more details.

And I guess that's it for the pydev configuration... code-completion, code-analysis, debugger, etc. should all be working.

You can now enjoy working with django inside of pydev!

Cheers,

Fabio

Pydev 1.2.4 released

This release fixes a number of minor things from the last release and adds some new features... I'm not sure if it was the alignment on the planets, but after only 5 days a considerable amount of stuff was done, so I think that a release was a good option (even without any outstanding bugs in the previous release)

Some noteworthy things:

- A long-open debugger bug was fixed with a patch from Kovan (on linux, sometimes it would not show variables after a step-over)

- Yield is correctly treated as an expression now for python 2.5

- Code-completion for parameters: in the open-source version it will get the completions for the tokens that are previously defined in a context and offer them as completions. In the extensions it will bring suggestions with all the tokens available for class methods/attributes for the project. Currently it shows all, but I intend to filter those later, to base it on previous tokens already declared in the context.

- Code-analysis: now won't give you errors if it finds:

• classes that have __getattribute__ or __getattr__ overriden


• assigns that are initially None


• assigns to a function call and not a class definition

Hope you enjoy!

Grab the extensions at:http://www.fabioz.com/pydev/

And the open-source at:http://pydev.sourceforge.net/

Pydev 1.2.3 released

Nice numbering hum?

It's out -- support for 2.5 right on time with the python 2.5 release... I didn't really plan for it like this, but I think it's a nice touch ;-)

In case you want the details for the release, check:

Pydev Extensions: http://www.fabioz.com/pydev
Pydev: http://pydev.sf.net

And I've covered most details on my previous posts, so, I'll leave this one short.

Support for python 2.5 finished!

Ok, the support for python 2.5 should be finished (surely there will be some things I'm missing, but it should be already ready for use... still, I'd currently regard it as unstable, as there were many changes in the core to make it work)

There are some things that will not go a 100% on the way python does things. Namely:

1. 'with' will always be highlighted as if they was a keyword, so, if you have code such as 'with = 1' it will be highlighted as a keyword even if you're using the grammar for python 2.4

2. If you say you're using python 2.5 in your project, it will give errors when:
- it finds the 'with' used in its old form (python would only warn you about it)
- it finds the new construct used without the 'from __future__ import with_statement' (that's the same behaviour python has)

3. Using the from __future__ import absolute_import will currently have no effect for pydev (so, it will still go after relative imports even if you specify it (unlike the ones specified before, I hope to change that in a later release).

There are some specifics to the pydev working also:
- It will now use for external files the same grammar that is specified for the actual interpreter, so, if your interpreter is for version 2.5, that's the version that it will use to do it (now and then I receive a message from people with custom interpreters, so, it is important that those interpreters have the sys.version_info return the correct info when setting up the interpreter)

- If you've specified the interpreter in a previous version, it will acknowledge it as version 2.4 even if it is actually version 2.5 (so, you have to remove and add your interpreter again).

I hope you enjoy it... it should be out in the beggining of the week... hopefully on monday or tuesday.

A little bit more from the parser

Ok, I think that the parser is working for all the changes in the 2.5 grammar

Adding the unified try..except..finally statement was only a change in the grammar, and it produces the same AST as if it was a try..finally with a try..except, so no semantic changes should be needed.

The new "with x: ..." statement was also added, but I still have not checked which semantic changes are needed. It will also need a new keyword in the syntax highlighting.

So, the parser is now almost complete... almost because now we have to mark those new constructs as invalid if the grammar is in 2.5 but you're using code in 2.4. Still, that should not be difficult to do.

Support for Python 2.5

No, it's not ready yet, but it has progressed a lot today.

Pydev now already has parser support for some of its new features that require changes in the grammar:
- The new relative import (from .. import foo / from ...bar import foo)
- The new if expression (a = 1 if bla else 2)

The code analysis seems to be working fine for the new if expression, but the new import will need additional work for recognizing it semantically (so, code completion still does not work with that new version).

Adding that new if expression was really tricky... Also, I always have to stare to the grammar and the javacc file for about 1 hour until it makes sense again ;-)

And here are some links needed to make those upgrades...
-- python grammar: http://svn.python.org/view/python/trunk/Grammar/
-- python parser (contains a version of Python.asdl for python: http://svn.python.org/view/python/trunk/Parser/)
-- what's new in python 2.5: http://docs.python.org/dev/whatsnew/whatsnew25.html

Also, it's a holliday here in Brazil, so, I won't be online again until monday...

Focus for next releases

I've been thinking about on what's the next thing to focus on pydev, and I think it'll be in usability, aiming for those starting up with pydev for the first time...

Many say that Eclipse has a steep learning curve... If you're a first-timer, it may be true if you don't try to get some help (the tutorial at fabioz.com is something you really shouldn't skip if you're using pydev for the first time)... still, I think that the learning curve to master pydev (and Eclipse) is not as steep as for mastering vi or emacs...

Also, I've seen many people go from vi/emacs to pydev (extensions) and be more productive in a short time (like a month or two, which is not so much for users that used vi/emacs for more than 3-4 years).

Ok, I guess I deviated a little from the topic ;-)

So, the focus for the next releases will be on usability and getting python 2.5 on pydev.

Pydev release 1.2.2

Yeap, version 1.2.2 it's already out.

Code-completion had special attention in this release. After using it, it's hard to imagine how did I program without those features for so long... easy to get used to some things hum?

Ok, the difference is not the same as not having the extensions code-analysis, but still...

Well, check the details at: http://pydev.sourceforge.net and for the extensions: http://fabioz.com/pydev

Improving code-completion in pydev

The next release is taking a bit longer than usual, but I think it'll be worth it ;-)

The last releases were pretty much focused on getting pydev more efficient -- with lots of profiling -- and stable, and in this next release, I'll focus on features again (mostly on code-completion).

Calltips is one of those long-awaited features that can make your life so much easier, and will be added in this release (and needs no presentations), but as this release is focusing on features, it'll also add:

Toggling the completion mode

That's one of those things that's hard to explain... But I'll try it:

When you're in a piece of code such as

foo.doIt(a, b)

and you want to do a completion to change the method 'doIt' (after 'do') for 'doItNow', it would usually do: 'foo.doItNow(xx,yy)It(a,b)', and you'd have to delete that part with '(xx,yy)It' after applying the completion. Now, in the new release, after bringing the completions up, pressing 'ctrl' (the key that toggles the completion mode) will replace the rest of the method for the new completion, and don't put the parentesis if there is one already.

Cycling through completions

Now, pressing Ctrl+Space multiple types will cycle through the default completions and templates completion (so, if you're looking for a hard-to-find template, just press Ctrl+Space one more time).

Linked Mode

After applying some completion, if it has parameters, you'll be able to cycle through the arguments with Tab and pressing Enter will exit this mode and place the cursor just after the call.

Configuring Pydev to work with Turbogears

I still haven't used turbogears myself, so, in the end, this might be interesting for other users that are starting in the turbogears world themselves ;-)

1st step: getting turbogears

OK: downloaded version 0.8.9, which seems to be the latest release

2nd step: installing it

Running setup.py install did some nice things for me... it downloaded setuptools automatically (as it was a pre-requisite) and then proceeded getting other dependencies (egg files) and installed all those without further problems (yeap, many dependencies there).

Now, I still don't have docutils (which it says will make it more 'fun') nor a database, so, I'll get it before I proceed.

It says to use easy_install to get docutils, but it needs some search to know exactly what it means (turns out to be a script installed in my computer at C:\bin\Python24\Scripts\easy_install.exe)

Turns out it didn't do it's job:

[C:\bin]c:\bin\Python24\Scripts\easy_install.exe docutils

Searching for docutils
Reading http://www.python.org/pypi/docutils/
Reading http://docutils.sourceforge.net/
Best match: docutils 0.4
Downloading http://prdownloads.sourceforge.net/docutils/docutils-0.4.tar.gz?download
Requesting redirect to (randomly selected) 'mesh' mirror
error: No META HTTP-EQUIV="refresh" found in Sourceforge page at http://prdownloads.sourceforge.net/
docutils/docutils-0.4.tar.gz?use_mirror=mesh

So, I went on to get it myself and then proceeded to get pysqlite (http://pysqlite.org).

Now that all the turbogears pre-requisites are there, let's see how pydev looks... First, my interpreter configuration has become obsolete, so, all those libraries that were added later are not there... To make it up-to-date, I have to remove the interpreter and add it again (so, it should get all those new libraries -- version 1.2.0 of pydev and earlier had a 'cache' bug, so, you'd need to remove the interpreter, press apply and only after it proceed to add the interpreter again).

Now, just to make sure pysqlite is there, create the following script to see if it works:
from pysqlite2 import dbapi2 as sqlite
con = sqlite.connect("mydb")
print con

Ok, connection there, and a mydb file created, so, everything seems fine.

I decided to go with the 20 minute-flash that turbogears has there...(now that I'm revising this, I'd advise against it and would go with the wiki tutorial, as the flash version hides too many things). It starts creating a project with tg_admin quickstart. So, I decided to create a new pydev project from eclipse before and placed it at d:\turbogears, and ran tg_admin quickstart from within that directory to get started.

I called the project "wiki test" and it created all the structure without any problems, and starting it with wikitest-start.py actually started the server without any problems (http://localhost:8080/)

Now, to configure pydev, you have to add the created folder d:\turbogears\wikitest as a source folder for the project (this would give you code-completion and code-analysis for that project). Details on how to do that can be found at http://fabioz.com/pydev/manual_101_project_conf2.html

Ok, now on to the database creation: got some things mixed up here because the video didn't specify the dev.cfg changes (but the written one does, so, not such a big problem).

Kid template editing: I'd reccomend having WTP (Web Tools Platform) to edit html files... surely makes things much nicer.

Interesting point: I've just now noted that turbogears spawns 2 shells when I run the server, I think that one is the http server and the other some auto-reloader, but this makes some things much harder:

1. If you run it from inside pydev and kill the shell from the eclipse console, the other process will not die (and as you don't have Ctrl+C in that console, you will have to keep killing the other shell manually).

2. You just won't be able to debug it from pydev, as that other shell will be the one that will actually serve the requests (altough you can do it with the remote debugger from pydev extensions).

So, after checking its code a little I've seen:

if conf('autoreload.on', defaultOn): ... do the auto-reload stuff, so, you can set autoreload.on=False in the configurations and run it from pydev, so that you can debug it (but this way, you'll loose the auto-reloader stuff, so, I guess you'll have to decide: pydev regular debugger without auto-reload or pydev extensions remote debugger with the auto-reload (in this case, you could even run it from outside of Eclipse).

So, I guess that's it. Pydev is configured up to the point where you can use code-completion, the debugger and code-analysis without any problems.

Cheers,

Fabio

Pydev 1.2.1 released

Actually, Pydev 1.2.1 has been released 2 days ago, but the final Eclipse 3.2 (which is the one that pydev targets) will be released later today, so, I think it's a good time if you were waiting to upgrading to the latest pydev release.

Information on the release for:
- Pydev Extensions: http://www.fabioz.com/pydev
- Pydev: http://pydev.sf.net

Details on its development: http://pydev.blogspot.com

Notes

Ok, just as a note to pydev extension users, http://www.fabioz.com/pydev will be out this weekend due to some restructuring in the network where the server is, but it should be back on monday.

And in the beggining of the next week, there should be a new release (1.2.1), with fixes for some integration issues with Eclipse 3.2 (the major one is that variables in the debugger did not show on 1.2.0) and some minor enhancements.

Pydev 1.2.0 released

Pydev and Pydev Extensions 1.2.0 have been released

This release focused mainly on making pydev compatible with Eclipse 3.2 (so support for Eclipse 3.1 is now discontinued) and optimizing pydev to make its operation 'smoother'.

Another nice thing is that the outline now has more info (attributes, and comments starting with #---)

More details can be found at:

Pydev Extensions: http://www.fabioz.com/pydev
Pydev: http://pydev.sf.net

Pydev 1.2 shaping up

Yeap... Pydev is going to the gym ;-)

I've spent the last week just evaluating it and making it run faster... I belive that it will have about a 20% gain in performance overall (depending on which features you use could be more or less).

Special attention was given for the parser, because it is used on lot's of things.

A (big) memory leak was also found and removed... I'll even comment a little more about it, as it was something that "amused" me for a while.

When I did an analysis, pydev would create a thread, and depending if it was a full build or a delta build, it would call 'run' or 'start' on the thread... Now, I've discovered that if you call 'run' in a Thread and not 'start', and loose its reference, it will NOT be garbage collected, so, you'll loose that memory forever (way to go ThreadGroup)... needless to say that this particular thread held a reference to the generated AST, so, basically, all the ASTs generated in a full build would be there for eternity -- considering eternity the lifetime of Eclipse ;-)

This is fixed for 1.2... Also, many changes were needed to make it work with Eclipse 3.2, so, please, wait for the next release of pydev to come out before migrating!

Pydev 1.1.0 released

Pydev 1.1.0 has just been released. It has a number of bug-fixes and enhancements (check http://pydev.sourceforge.net/)

This release will be the last release targeting Eclipse 3.1 -- the next release will be targeting Eclipse 3.2.

Pydev faster start-up

Ok, when starting pydev, until now, all its aspects where initialized linearly when some part of the plugin was requested. As you usually started with the editor, the time it would take pydev to be usable was about 40 secs (in a good machine), because it would have to start:

1 - the plugin (major work was on loading the interpreter information)
2 - the editor itself (major work was on starting up the scripting engine and binding some actions to the editor)

-- This was just the initial loading time (one-time operation), so, new editors after that would be pretty fast, but still, that initial start-up could be annoying sometimes...

So, now the structure was changed so that you can start pydev and begin using it making the other initializations in a thread. It became a bit more complex, because binding actions would have to sync with the main thread and so on, but on the other hand, after starting pydev, it would take (in a good machine) 1-2 secs to open an editor that first time (quite acceptable in my opinion).

When that happens, some actions might still not be 'readily' available, such as the scripts, so, if you request them right away, they just won't be available as you'll have to wait those other threads make their work, but still, you can already open and begin typing without using that (which is the most common behaviour).

I also think most performance bottleneck were sorted out, so version 1.0.9 should be quite responsive -- it should be released in the start of the next week.

Cheers,

Fabio

Pydev and SoC 2006

It's official: Pydev will have a student working on Google's Summer of Code!!

Sean Handley will be working mainly on some debugger features and in the code-completion engine, doing some long time requested features.

I'll be mentoring him, so, we'll see how that goes -- I'll keep you informed ;-)

Sean will be working on a branch so that we can manage to work both without stepping on each other's toes, and I'll synch that branch to the HEAD from time to time. I know it'll probably be more work, but I hope it will be easier to manage things this way.

It was also pretty nice to see that pydev had a good support from the PSF (otherwise it would surely not get a project approved at the SoC -- and even better: it was among the top of those approved projects, which is quite an achievement given the number of good proposals submitted).

So, that's it... let the work begin ;-)

p.s. It's something very strange to me, being a part of the SUMMER of Code, since it's damn cold (at least for me) here in the south of Brazil! ;-)

Pydev Release 1.0.8

Pydev 1.0.8 has ben just released... A major bug triggered this release (that's why it's been issued in less than one day from the previous release).

Mainly, if you had a file that had a docstring at the global level with an empty line, it could get to a loop when adding a new-line to the document. This has been fixed and is already available for download.

Also, 2 other minors have been done for Pydev Extensions (but they surely would not be worth a release for them).

-- Fabio

Pydev release 1.0.7

Ok, the new release is out already.

This release focused mainly on bug-fixing and optimizations, so it had only few features added. This is actually a sort of release candidate for 1.1, that will be a major release, and will be the last release to support Eclipse 3.1 (after 1.1, releases will support Eclipse 3.2).

In pydev the only feature added was a contribution by Don Tailor with the Jython Scripting Engine (if you would like to create your own, you can take a look at http://www.fabioz.com/pydev/manual_articles_scripting.html)

And in pydev extensions, a mark-occurrences and rename-occurrences was added (as well as some options to make remote debugging more attractive).

As always, you can check details on http://www.fabioz.com/pydev and http://pydev.sourceforge.net/

Cheers,

Fabio

Pydev release 1.0.7

Ok, the new release is out already.

This release focused mainly on bug-fixing and optimizations, so it had only few features added. This is actually a sort of release candidate for 1.1, that will be a major release, and will be the last release to support Eclipse 3.1 (after 1.1, releases will support Eclipse 3.2).

In pydev the only feature added was a contribution by Don Tailor with the Jython Scripting Engine (if you would like to create your own, you can take a look at http://www.fabioz.com/pydev/manual_articles_scripting.html)

And in pydev extensions, a mark-occurrences and rename-occurrences was added (as well as some options to make remote debugging more attractive).

As always, you can check details on http://pydev.sourceforge.net/

Cheers,

Fabio

Pydev package explorer

No, currently no plans for it, but there are some good news...

One of the major reason pydev has no navigator is that eclipse still does not provide a way to do it easily, you'd have to create it from ground up, with the TreeViewer, etc.

Now, it appears that version 3.2 will have a commom base for creating it...

So, I decided to google for "eclipse Common Navigator" and got some nice results, which I'm posting to remind me of checking them once the Eclipse 3.2 final is released (and based on this results, I believe that the package explorer will be available shortly after that final release of Eclipse 3.2).

A good example on how to use this api can be found at:
http://scribbledideas.blogspot.com/2006/05/building-common-navigator-based-viewer.html

-- Fabio

Remote debugging getting smarter

There're 2 things that can get in the way when remote debugging some application (especially if it's in another computer):
- Not being able to find the source code
- Not being able to get the output

So, I've changed the debugger in Pydev to make it smarter to find some file if it does not have a 'direct' corresponding file (so, if in the server you have a file mod1.py in a path /usr/local/foo, you'll be able to specify that location in your computer -- say c:/projects/foo/mod1.py). Also, if it has only one file named that way in the workspace, it will open it directly (and if more are available, it will give you a choice over which file to open), and as a last resort, if it is not found in the worskpace, it will allow you to choose a file in the filesystem.

As for getting the output, 2 new parameters were introduced in the settrace function, so that you can specify: pydevd.settrace(stdoutToServer=True, stderrToServer = True). Through that, it will redirect all that goes in stdout and stderr to the debug server. Note that I've done it in such a way that it will still print it in the original stdout and stderr -- so that you don't loose your logging (if you have any).

This will be available for the Pydev Extensions release 1.0.7 (which should come out by monday or tuesday).

-- Fabio

Pydev Videos

I've done 2 videos about pydev. One is pretty close to the getting started tutorial, and explains how to configure pydev and use it to the point where you can debug some module and the other explains some 'power' features for pydev and pydev extensions such as code-completion, navigation utilities, code-analysis, quick-fixes...

Both have a little more than 6 minutes. They are hosted at showmedo.com and are recorded with voice (I guess you'll be able to tell that english is not my first language -- but still, I believe you should be able to understand it without any problems).

What's next?

Ok, I'm getting the new pydev release ready to go... There have been a number of bug-fixes, so 1.0.7 should be out soon.

I'm trying to get the 'mark-occurrences' feature out in the next release, but I'm not sure if I'll get it complete in time. Still, I might release it early (in a beta-state) if it turns out it'll take too long, as there have been many bug-fixes and speed enhancements that I think are already ready for the release, and it is not a critical thing, so, you can simply disable it if you want.

Another thing is that the sf cvs is (finally) up again, and, I've just synched the developers cvs with the last version (I'm not sure how long it'll take to sync with the non-developers cvs).

-- Fabio

Profiling Pydev

The first thing I profiled was the grammar used, since it is used among many places, I believe it to be the something that would have a good impact overall.

One of the things we end up doing when testing some stuff is embeeding some file -- say, an image -- into a .py file. Among those, we could get a 3-4 mb sized file, and pydev would take a long time to parse it, so, my first target was to speed that up.

Making 'general' changes thinking it will get better is usually a mistake, as you need a way to measure those impacts. I started then with a file with a couple of statements and a single 'huge' multiline string and made my target optimizing parsing that file faster.

After playing around a little in the parser I discovered that the actuall speed loss was not at the grammar itself, but at the Reader that should give the chars to the tokenizer. After looking at its code, you could see it allocateded lots of memory in the process, so, I decided to create another reader from scratch and with the help of some unit-tests, and the results were pretty impressive (for big files):

Parsing a huge .py file (3mb) it was taking about 4-5 minutes... now it only takes 2-3 seconds (yeah, the previous approach had an 'exponential' behaviour depending exclusively only on the size of some file, not to mention that it would make the garbage collector work a lot more).

This will be available for 1.0.7 -- But before I do release it, I'm still looking for other 'hotspots' to optimize ;-)

Fabio

New release (1.0.6) and targets for 1.1

Ok, release 1.0.6 is out. I hope you enjoy it. This one had only few features added, and centered more on bug-fixes.

There was one bug found that should be corrected by those that got version 1.0.6. It is a simple fix, but could save you some headache until the next release comes out:

Go to org.python.pydev/pysrc/importsTipper.py -- line 191 on:
doc = inspect.getdoc( obj )

and make
try:
doc = inspect.getdoc( obj )
except:
doc = ''

Yeah, didn't know that could throw an exception (I guess that's one of the difference between checked vs unchecked exceptions... Altough I surely don't miss declaring them in python -- even in java, I'm currently using almost only RuntimeExceptions, and only make checked Exceptions on a few selected cases).

This fix is because when trying to get some info on compiled modules, inspect.getdoc could go nuts ;-)

So, basically the target for 1.1 will be on making what pydev has so far as stable as possible, as well as adding support to Eclipse 3.2. So, probably in the next 1-2 months, the releases should be centered on bug-fixes and profiling (to make what we have faster...), and maybe some minor features.

I'll probably have to swell onto the grammar, as I believe that's one of the most used parts of pydev...

-- so, does anyone have any decent profiler to reccomend? I've already evaluated JProbe, which I thought did a good job (I've been able to do some nice enhancements based on its reports), but did not buy it after the evaluation period ended (I've gone to their website and was not even able to see how much it was... I've found a 'buy' link at the top of the page, but it didn't show info about JProbe, and then after about more 5 minutes browsing in their page to find how much it was my patience ended and I decided to go looking for something else).

Hierarchy view

I've been playing with a hierarchy view recently, and got some interesting results already... In python an interesting aspect is that you have multiple inheritance, so given some class you can have a Tree of superclasses as well as a Tree of subclasses.

So, instead of using a Tree View as JDT does, I decided to try something such as showing it similar to a class diagram. After a long search in the net for 2d drawing toolkits, I've decided for Piccolo. It's a nice library, and I've had only few problems using it. A point that favored it a lot over other choices is that it supports SWT natively, without the need for any other library, and besides, its abstractions are pretty easy to follow.

The results I got so far are in the image below...



Note that the graphic part is actually the 'easy' part... doing a search for references (to get the subclasses) is the difficult thing, but it seems to be shaping up nicely. I hope to release it (at least in a beta state) for the next release of pydev extensions.

Debugging the pydev debugger

Yeap, the debugger keeps appearing now and then... I guess it shows that many people use it ;-)

So, this time the updates are:

1. Not caching all frames: This is mostly because the debugger currently stores all the frames in a cache, but not as weak-refs (you can't make a weak-ref to a frame), and this could make some applications not work as expected. So, that's just been fixed: pydev keeps a weak-ref to the thread and adds the 'last' frame to that thread, so, if the thread dies, the frame should die too, and we will only have one strong reference to the executing frame (which can be used to find its parents when needed).

2. Synchronization: Yeap, the debugger NEEDS to handle multi-threads, so, this is something pretty important, as some halts were reported before... while I've not been able to reproduce those halts, I believe that the synchronization plus not keeping frames (and therefore all the locals) alive SHOULD solve some halting problems reported (it is always difficult to solve something you can't actually reproduce, especially in highly complex things as the debugger).

3. The F7 (step-return) now stops in the caller call for something (instead of stopping in the next line). This one is hard to explain, so, let' me give an example:

call2(call1())

Entering call1, and making a step-return, would make it miss call2 and stop on the next line of code. Now it stops in the last line of the call1 context, enabling you to keep on stepping into call2 (you could do that before, but you'd have to know you were in the last scope and make a step-in, so, sometimes you could miss that and it could be pretty annoying).

So, that's it for what's going on in the debugger... I'll leave other news for the next post ;-)

Release 1.0.5

Ok, 1.0.5 was released yesterday... I've released it again in such a short time because the indentation engine had some minor quirks (and when it is something you use as much as indenting, even little things can become a major nuisance).

And besides, there were some other nice things such as being able to add vm arguments when running a module and a way to kill the python shells pydev has running for code-completion/refactoring purposes.

Most people probably will never use the 'kill shells' feature, but if you're doing compiled modules, you might be interested... When you're doing compiled modules, you'll not be able to link the library if it is currently being used from a python shell (as the code-completion shell), so, this was added so that you don't need to use the task manager (in windows) or to a shell in unix to kill those processes.

Pydev Extensions didn't have any major change -- I did only some minor changes to the time the pop-up pops for non-licensed versions -- removing it altogether or making it softer than what it is now beats the whole concept of having it anyway... Which as most should know is not annoying users, but raising money to keep pydev (open source and commercial) alive... So, if you don't like it, you can either license it (and see it vanish) or stay with the open source version -- both versions have been growing a lot lately ;-)

Fabio

Pydev Release 1.0.4

Just released!

The 'major problem I had in this release is that the cvs at sourceforge is down (for some days already), so, I had to set up the code in a local cvs, and will sync it as soon as they the service is up again. Unfortunatelly, they have no dates or estimates on when will it be fixed (and so, the code will not be available until then).

On the good side, this release had some major features for both versions.

The major features added are:

- In Pydev Extensions, you can have a shell binded to the editor (and I dedicated my 1st screencast to it: http://www.fabioz.com/pydev/manual_screencasts_presentation1.html)

- In the Open source version, there is the new indentation engine and the possibility of scripting pydev with jython (I've created an article for it at: http://www.fabioz.com/pydev/manual_articles_scripting.html)


So, I'm expecting feedback on the scripting engine. Also, I intend on doing some place so that anyone can contribute its own scripts to pydev -- currently, pydev-code is the place to go for that, and for help on how to create your own scripts.

Hope you enjoy,

Fabio

p.s. I've also changed some things in the layout of the blog -- mainly, the relevant links where put in the start of the page, as well as the RSS link (which was something that popped up once in a while).

New release is coming...

Ok, as usual, there are lot's of things happening in the Pydev world, so, let me tell you what you can expect for the new release...

1. Indentation: Just 1 word: 'Wow'
-- I don't even recognize it anymore ;-)

2. Jython scripting: you'll be able to script pydev with Jython (I've already wrote a longer post about this one, so, I'll keep it short).

3. Pydev Extensions got itself a console that you can use integrated with the editor (so, you can have all the interactive stuff from a prompt, and the editing features of Pydev) -- and you can always use it as a regular console if you want.

4. This one is a minor compared to the others (but still, noteworthy): Ctrl+. gets you to the next problem marker found in the document.

So, those are going to be the major features for the next release... I expect to release it next monday, so, keep tuned.

Eclipse FAQ published

Ok, this is just a post to remind me of a good place to find documentation when coding Eclipse.

http://wiki.eclipse.org/index.php/Eclipse_FAQs

Pretty Nice (and thanks to John Arthorne and Chris Laffra for writing it)

Scripting Pydev with Jython

Ok, the new release already has its 'feature-set' defined, and among its new features is the possibility of scripting Pydev with Jython.

-- This will be added to the Open Source version.

I've already implemented a first example, and it seems to work ok. You can check it at this link.

Basically, I create a PythonInterpreter (from jython) that keeps listening to the Editor (so, it receives notifications of setting actions, document, disposing, etc). That script is always called when a notification is sent, with some 'locals' that are put, such as the editor that launched the notification and the command received.

For the integration, all the scripts in the jysrc folder that start with 'pyedit' will be compiled and ran. And if you change it, it will automatically 'know' that it changed and will reload it... pretty neat ;-)

In the example, it only cares about the 'onCreateActions' notification, and binds an Action to an activation code (in the case, Ctrl+'.').

I know it is still a simple example, and the scripting engine has a long way to go (I didn't even test if more than one script would be able to communicate to the other, nor did I add the jython Lib to the path), but should be a good initial step for that integration ;-)

Also, making code-completion in Jython with Eclipse internals is something that I still have to define better (it would be insane adding all the Eclipse jars to the project). Hopefully, in the future, you'll just be able to say that it is a plugin project with Jython files, and it will be able to get all that (until then, all the imports will be undefined while editing it, so, no code-completion for it)... On the other way, you can already have code-completion for Jython-related stuff if you configure your project correctly.

Cheers,

Fabio

New Pydev release: 1.0.3

Ok, this one contains some 'amenities' in the editor (in both versions):

- You're able to change the color and style of class and function names
- There is auto-dedent on elif and else

And in the Pydev Extensions version:

- added an 'open declaration' quick outline... (I was REALLY missing that when programming in python).

Basically, it opens a dialog with all the classes, methods and attributes (even if not top-level) and shows all so that you can go directly to some declaration (the keybinding is Ctrl+Shift+T -- and you need focus on the Pydev Editor).

So, enjoy your new features!

Cheers,

Fabio

p.s. some bugs were fixed too ;-)

Pydev Extensions

Wow, just now I realized that I still haven't posted about it... which is a real shame! ;-)

Well, Pydev Extensions (http://www.fabioz.com/pydev/) is a commercial plugin dedicated to extending Pydev (and adding those 'power-features' that can make your life so much easier). It has many new features, and has a lot of development going on.

The most 'noteworthy' (and already very mature) features are:

* Code completion with auto-import: I don't know how I ever managed to write code without it!!

* A 'real time', embedded code-analysis. This one is pretty nice, as it also allows getting errors while you're writing.

* Quick fixes for the errors found... From these, I guess that the most noteworthy 'quick-fix' (activated with Ctrl+1) is the one that allows you to automatically make an import on some undefined token.

* A go to definition that is a replacement to the one provided by BRM (again, I don't know how I managed without it before...)

* A debug server (allows debugging programs not ran directly from pydev) and a debug console (so that you can inspect things easier when you hit a breakpoint).

Well, there are more features... you can find the complete set at: http://www.fabioz.com/pydev/manual_adv_features.html

Also, it has *almost* a full month of existance -- which is still being celebrated -- And has lots of users already!

You can grab it and use it for 30 days without a license, so, ENJOY!

Cheers,

Fabio

p.s. This week we have 'carnival' here in Brazil, and I'll probably only be back on Tuesday... so, your e-mails and posts might take a little longer than usual to be answered.

Test Driven Development in Pydev

I'm currently doing the pretty-printer for pydev, and was thinking about the way I do things...

In the end, my 'usual modus-operandis' can vary somewhat depending on what I'm developing. Usually, I like to do things with tests up-front, as was the case for the most intricate parts of pyde, like code completion, code analysis...

On the other way, for things I consider simpler, and that usually have a good 'visual' feedback, such as populating the outline, I must say there's little testing I do... basically, I just 'wish' very hard that it will keep working. I guess I must be a 'good' boy, as usually these kind of wishes come true to me ;-)

Anyway, being 'test-infected', sometimes I wonder if I should test more these interactions... I basically think that it wouldn't do much difference for those simple cases, as the time I would need to spend writing those tests are probably higher than the time I spending fixing these bugs (pydev has a 'low' number of bugs for these things... and I'm not even sure if adding those tests would lower those numbers).

On the other way, I think that it would be 'almost' impossible to deal with the critical parts without it... Let's take code completion as an example. If someone takes a look at its implementation, you can see that there is a lot of dependencies among modules, and a lot of recursivity going in the way that tokens are resolved, and much more than once, when I was up to fix something I ended breaking something else that had a dependency on that, but thankfully -- I would say luckilly, but then I thought better... luck has nothing to do with it ;-) -- the tests kept that 'error-cycle' very short.

So far so good right? Anyway, I guess that the only 'bad' part about the 'test-infected' way of doing things is sometimes not taking the time to try to refine my initial design to make some things easier from the beggining... I mean, probably in the end, with the tests and refactoring, the result could be very close, but there's a 'middle-ground' somewhere between those, which I feel would be the optimal point.

On the other side, I must say that in pydev, many 'complicated' things, didn't even have that initial design, some because I didn't 'feel' like doing it and some because I had no idea where to start, so I just wrote a test and made it pass... and then other... and other ;-)

Checking how to do code-formatting

I was giving a quick look on how could the code-formatter be better, and having researched a little on how would it do the 'pretty-print' I found some references:

Dealing with comments:

Approach 1 (using special token):
- https://javacc.dev.java.net
/servlets/ReadMsg?list=users&msgNo=227

Approach 2 (not using special token and actually putting comments in the parsing.
- http://cvs.sourceforge.net/viewcvs.py/jrefactory
/JRefactory/javacc/java1_5.jjt?rev=1.18&view=markup

Currently, I generate an asdl structure that would also need to be extended to be supported later, but I guess I'll have to work on this myself, as I could not find any reference about comments in asdl... anyone has any suggestions?

And a little 'offtopic' but also for javaCC:

Error recovery: http://www.cise.ufl.edu/help/software/JavaCC/errorrecovery.html
Getting special tokens: https://javacc.dev.java.net/doc/tokenmanager.html

Cheers,

Fabio

Pydev 1.0.1

Yeap, just some hours after releasing version 1.0 I had to release 1.0.1... This is because of one of the stupidest bugs I ever did...

I accidentally commented a line... and it probably occured between passing the unit-tests and making the build.

Yeah,yeah, I know, I should pass the tests only when it is already built... but I still have to work out on how to do that in the pydev structure.

Well, 1.0.1 is already out, so, enjoy it.

Cheers,

Fabio

Pydev 1.0

Well, that's it, the next version will be 1.0. Pydev has surely already reached its maturity. I will try to release it next week.

There were tons of improvements over the last version, in the debugger, in the PyLint integration, wizards, perspective, shorcut to run the editor file, etc.

Also, next week I'll start giving commercial support to Pydev and release a new plugin named 'Pydev Extensions' -- which has already quite some time of development (and in that, I hope to be able to give better support for the Open Source version too).

I'll announce what it can do next week, together with the 1.0 release of Pydev... But just as a little 'preview', think about faster code analysis (not PyLint), auto-import, remote debugging... ;-)

Actually, I've been thinking about that for quite some time now, and I think this is going to be a win-win situation in many ways... Hope you all enjoy it too...

Cheers,

Fabio

Debugger (again)

Hi All,

I think I'll stop playing with the debugger... it's not fun anymore... There's nothing left to speed ;-)

I've tried some other python debuggers, just to compare them in 'performance', and I must say that pydev took an A+ (considering that those are A+ too!!)

I won't say the names of those, because you never know what others could say about it, but in speed terms the new version 0.9.8.8 (still unreleased), will be the best by far when compared to other pydev versions.

I've also managed to make the variables be returned 'on-demand', so that there are only 'very-short' delays when hitting a breakpoint and for later getting any variable (instead of a very long one when a breakpoint was hit).

As for features, we've already have the basic stuff such as breakpoints, step in, out, variable inspection and some intermediate stuff, such as evaluating some expression or making a conditional breakpoint (and all that supporting python and jython).

I believe that what is missing for a 'complete' feature-wise debugger is:
- break for exception (intermediate)
- pause running program (intermediate)
- remote debugging (intermediate): this would require the user to add pydevd.py in its pythonpath and use some calls to it (I already have a 'prototype' for it, but won't make it available just now...)
- attach to running program (hard) note: this is quite different from having just a 'pause' button (in which case we already lauched it from within pydev), and I'm not really sure how feasible it is
- mixed-mode java/jython debugging (hard)

Cheers,

Fabio

Debugger speed (up and up again)

Yeap, I decided to play a little with the debugger again... It is strange as sometimes just a 'little' profiling can speed things a lot more...

Now, this time, I did 2 improvemens:

1. The way that the main 'dispatch' was being handled: every time a new dispatch was called, I got the file information again from the frame (and had to do a couple of os.path calls), but in the end, the overhead for that was huge, so, now I 'cache' the context so that the filename and the base of the filename are stored in a cache.

2. The way that the shell was communicating with the shell was in the 'main' loop, for each call... now it is in a thread (I thought that this was going to be the major speed up, but number 1 was much more effective at it).

That will be available at release 0.9.8.8

Cheers,

Fabio

Release 0.9.8.7

Just released it... There are many changes:

The debugger does not print its tracing anymore, plus some problems with the jython shell and other bugfixes.

The actual 'feature' for this release is that the interpreter configuration will be backwards compatible from now on, but this has a drawback in the release, meaning that the current interpreter configuration will be lost.

Actually, I had to work on its structure and it became backward-incompatible, so, I decided that it's about time that it does not change anymore, even if its internal structure changes, and put it as a feature... actually, it just stores the information that the user enters in a place separated from the actual structure, which will be re-created with that info if it is not able to recreate by itself.

Cheers,

Fabio

Debugger problem in 0.9.8.6

Hi all, just noted that version 0.9.8.6 has shipped with a problem in the debugger...

The problem is that the tracing is on (which can make debugging MUCH, MUCH slower, because it prints many things to stderr).

To fix it, go to the plugin org.python.pydev.debug and change the file pysrc/pydevd.py setting "pydevd_trace = 0".

I'll try to release a new version on monday with that fix!

Cheers,

Fabio

New Release (0.9.8.6)

Hi all,

Quite some time without a release hum?

Well, it's already out... its changes are mostly bugs and some features (Thanks to Mikko Ohtamaa for those: new project and new module...)

Also, I'll try not to let you without a release for so long again, but in the last 2 months I had only little time to spend on it.

Hope you enjoy.

Cheers,

Fabio

What's going on...

Well, lot's of things happening right now...

I've got a hand from Mikko Ohtamaa, which has implemented some long-awaited wizards (create new python project and create new python module)... It still has some minor adjustments to do, but it's almost ready for release...

And another thing that might also make it to the next release is a package explorer (only in plans currently, but who knows...)

As for me, I'm currently trying to solve some problems that have been happening when using the code-completion shell for some users... I think I got it right this time... it sometimes got in a lock condition because of some threading issues... which for some reason never happened to me ;-)

Well, this release is taking longer that others, but I hope it is worth it!

Cheers,

Fabio