Pydev Release 1.3.12 and bug-reporting changes

Pydev 1.3.12 is out!

The major change is the speed-up in operations that require type-inference (e.g.: code-completion, code-analysis, etc.) and some bug-fixing!

Aside from that, from now on, bug reports will need a valid user registered at sourceforge... the main rationale for this is that most of the reports that come from anonymous users are usually not properly fixed because lots of times the 1st report misses the basic info to reproduce the bug, and if reporters can't be reached those end up being pretty much useless...

Eclipse Jobs API and Pydev Optimizations

1st thing to learn: Don't use UIJobs unless it's imperative that you have access to the UI... From 1.3.9 to 1.3.10 I was doing some refactorings to use the Eclipse Jobs API instead of raw Threads...

After taking a look at the core options (Job, UIJob and WorkbenchJob), the WorkbenchJob was chosen, as things shouldn't run if the workbench was shutting down (the docs say that it'll do checks to see if the Workbench is running, before actually running the Job -- as the Job in case: code-analysis is not suitable to run at shutdown, it seemed a nice choice)

But the thing is that the workbench job ends up running in the UI Thread -- and thus gives a little halt to the user every time the job runs... (and only after going after it in a bug report I found out about it).

On the good side, I took some more time to profile the code-analysis / code-completion (mainly when interacting with large files) and was able to make some nice improvements that will surely be noticeable... even on smaller cases:

- a cache that works for the duration of a type-analysis request to hold non-direct tokens available at a module was added
- a cache to hold scope information within a module was also created...

In my tests it gave a 40x improvement on a type-analysis request, as many traverses of the ASTs are not needed anymore with those caches (those will have a higher impact when a given module is imported a lot from different places).

All in all, optimizing Pydev is becoming more and more challenging :)

p.s.: Those improvements will be available on Pydev 1.3.12 (which hopefully will be released in the beginning of the next week).

Pydev release: 1.3.11

This release fixed some problems on the 3.3 integration (such as the undo/redo limit not being respected and the new 'insert spaces for tabs' in the general preferences conflicting with the pydev options).

Some niceties, such as being able to define custom filters in the pydev package explorer, not collapsing things in the outline unless really needed and having spell checking (which still depends on JDT -- because the Eclipse platform will only have that when 3.4 is released) are also there...

Now, the actual highlights were on the jython side this time: Java projects can now be referenced and have code-completion and go-to-definition working (actually, the go-to-definition is only available in pydev extensions) and the jython debugger should be finally working... There's even a little story about it:

The debugger used to work strangely in jython -- almost randomly... After lots of investigation, it turned up that pydev had a thread that was started with a delay to keep checking events in the debugger. That thread tried to run untraced, and changed sys.settrace(None) -- at that time, all stopped working (easy to say that now, but I had to go, get jython, compile it and run the debugger in a 'mixed mode' with the JDT/pydev debugger to find out about that).

In the end, changing sys.settrace in jython didn't only change the current thread tracing, but for the whole system... Paul Jean had already submitted a patch (which was applied to the trunk yesterday) so that that threads can run untraced... meanwhile, the pydev debugger is running with all threads traced on jython (checking for the actual version, so, as soon as a new jython release comes out, it should already be working).

Enjoy ;-)

Integration with java projects

The only thing keeping me from doing a new release is having the integration with JDT finished.

Pydev is already able (in the cvs version) to deal with java projects in a basic level (code-completion is there), but some things as find definition (among others) are missing.

Currently when a request for getting the completions is issued, a wrapper to manage referenced java projects is created (and that reference will be kept until the completion finishes) -- that's done in ProjectModulesManager#addModuleManagers

That 'modules manager' for java (namely: JavaProjectModulesManager) is able to create the java modules for classes it's able to resolve in that project: JavaProjectModulesManager#getModuleInDirectManager (modules that come from the system configuration are discarded -- that's handled in the jython system configuration already)

In that structure, each java class is internally mapping to a jython module and the completions are gathered from those modules -- those completions are transformed into ITokens to fit nicely the internal structure:
- AbstractJavaClassModule: has a collector for getting the completions and transforming them to ITokens;
- JavaClassModuleInProject: uses that collector for getting tokens at a java project;
- JavaZipClassModule: uses that collector for getting tokens in the system jars.

Yesterday I spent most of the day setting up a unit-test so that I could test those things (JavaClassModuleTestWorkbench), but now that it's done, I hope things will flow better in that integration (those JDT APIs are not really done for use outside of the JDT structure, so I've spent lot's of time to make it work -- and I'll probably spend some more time to make it work 'right'...)

Some of the doubts I had in the process:
JDT code-completion for external jar in non java project: Implemented at: JavaZipClassModule

API to find out about available JDT packages/classes in a project: Implemented at: JavaClassModuleInProject