Friday, January 10, 2020

PyDev 7.5.0 Released (Python 3.8 and Cython)

PyDev 7.5.0 is now available for download.

The major changes in this release are Python 3.8 support and improved Cython parsing.

Python 3.8 should've been in 7.4.0 (but because of an oversight on my part during the build it wasn't, so, this release fixes that).

As for the Cyhon AST, Cython is now parsed using Cython itself (so, it needs to be installed and available in the default interpreter for PyDev to be able to parse it). The major issue right now is that the parser is not fault tolerant (this means that for code-completion and code-analysis to kick in the code needs to be syntax-correct, which is a problem when completing for instance variables right after a dot).

Fixing that in Cython seems to be trivial (https://github.com/cython/cython/issues/3303), but I'm still waiting for a signal that it's ok to add that support to make Cython parsing fault-tolerant.

Enjoy!

Wednesday, March 27, 2019

PyDev 7.2.0 released

PyDev 7.2.0 is now available for download.

This version brings some improvements to the debugger and a fix for when PyDev could not properly find pipenv which could impact some users.

See: http://pydev.org for more details.

Friday, November 09, 2018

PyDev 7.0 (mypy, black, pipenv, faster debugger)

PyDev 7.0 (actually, PyDev 7.0.3 after some critical bugfixes on 7.0.0) is now available.

Some of the improvements available in this version include:

Mypy may be used as an additional backend for code analysis (see the preferences in the Preferences > PyDev > Editor > Code Analysis > Mypy). It is is similar to the PyLint integration, and will run Mypy whenever a file is saved in the editor.

Black can be used as the code-formatting engine (so, it's now possible to choose between the PyDev formatter, autopep8 or Black).

pipenv may be used for managing virtual environments (so, when creating a new project, clicking to configure an interpreter not listed will present an option to create a new interpreter with pipenv).

Improvements managing interpreters: it's now possible to manage interpreters directly from the editor (so, for instance, doing ctrl+2 pip install django will use pip to install django in the interpreter related to the opened editor -- and it's possible to change pip for conda or pipenv too).

Debugger improvements

The debugger is much faster for Python 3.6 onwards (when cython compiled extensions are available).

The performance improvement on the debugger is due to the using the frame eval mode for breakpoints again.

To give some history, that mode was previously disabled on pydevd because it had some issues which made function calls much slower (even though line stepping was zero overhead), but in the end, that overhead could make the performance worse than just using the plain tracing mode. Also, it had a big memory leak and in the cases where the tracing mode had to be reenabled both modes would be active at the same time and performance would suffer quite a bit.

-- all of those should be fixed and it should now perform better or at least on par with the tracing mode with cython in all scenarios... if you like numbers, see: https://github.com/fabioz/PyDev.Debugger/blob/master/tests_python/performance_check.py#L193 for some benchmarks.

p.s.: the improvements in the Debugger were sponsored by Microsoft, as the PyDev Debugger is used as the core of ptvsd, the Python debugger package used by Python in Visual Studio and the Python Extension for Visual Studio Code.

 

Monday, September 03, 2018

PyDev 6.5.0 (#region code folding)

PyDev 6.5.0 is now available for download.

There are some nice features and fixes available in this release:
  • #region / #endregion comments can now be used by the code-folding engine.
  • An action to easily switch the default interpreter is now available (default binding: Ctrl+Shift+Alt+I -- note that it must be executed with an opened editor).
  • It's possible to create local imports from global imports (use Ctrl+1 on the name of a given global import and select "Move import to local scope(s)" -- although note that the global import needs to be manually deleted later on).
  • The interactive interpreter now has scroll-lock.
  • The debugger is much more responsive!
See: http://www.pydev.org for more details.

Monday, August 13, 2018

Profiling pytest startup

I'm a fan of pytest (http://www.pytest.org), yet, it seems that the startup time for running tests locally in the app I'm working on is slowly ramping up, so, I decided to do a profile to see if there was anything I could do to improve that.

The first thing I did was creating a simple test and launching it from the PyDev (http://www.pydev.org/) profile view -- it enables any launch done in PyDev to show its performance profile on PyVmMonitor (https://www.pyvmmonitor.com).

Note that this is an integration test that is starting up a big application, so, the total time just to startup all the fixtures which make the application live and shut down the fixtures is 15 seconds (quite a lot IMHO).

The first thing I noticed looking the profile is that 14% of that time seems to be creating a session temp dir:


After investigating a bit more it seems that there is a problem in the way the fixture used make_numbered_dir (it was passing a unicode when it should be a str on Python 2) and make_numbered_dir had an issue where big paths were not removed.

So, pytest always visited my old files every time I launched any test and that accounted for 1-2 seconds (I reported this particular error in: https://github.com/pytest-dev/pytest/issues/3810).

Ok, down from 15 to 13 seconds after manually removing old files with big paths and using the proper API with str on Py2.

Now, doing a new profile with that change has shown another pytest-related slowdown doing rewrites of test cases. 


This is because of a feature of pytest where it'll rewrite test files to provide a prettier stack trace when there's some assertion failure.

So, I passed --assert=plain to pytest and got 3 more seconds (from 13 down to 10) -- it seems all imports are a bit faster with the import rewrite disabled, so, I got an overall improvement there, not only in that specific part of the code (probably not nice on CI where I want to have more info, but seems like a nice plus locally, where I run many tests manually as I think the saved time for those runs will definitely be worth it even with less info when some assertion fails).

Now, with that disabled the next culprit seems to be getting its plugins to load:



But alas, it uses setuptools and I know from previous experience that it's very hard to improve that (it is very greedy in the way it handles loading metadata, so, stay away unless you're ok in wasting a lot of time on your imports) and the remainder of the time seems to be spread out importing many modules -- the app already tries to load things as lazy as possible... I think I'll be able to improve on that to delay some imports, but Python libraries are really hard to fix as everyone imports everything in the top of the module.

Well, I guess going from 15 s to 10 s with just a few changes is already an improvement in my case for an integrated tests which starts up the whole app (although it could certainly be better...) and I think I'll still be able to trim some of that time doing some more imports lazily -- although that's no longer really pytest-related, so, that's it for this post ;)

Friday, July 06, 2018

PyDev 6.4.3 (code formatter standalone, debugger improvements and f-strings handling)

The latest version of PyDev is now out...

Major changes in this release include:

1. Being able to use the PyDev code formatter as a standalone tool.

To use it it's possible to install it as pip install pydevf (the command line is provided as a python library which will call the actual formatter from PyDev -- see the README at https://github.com/fabioz/PyDev.Formatter for more details on how to use it).

The target of the PyDev formatter is trying to keep as close to the original structure of the code while fixing many common issues (so, it won't try to indent based on line width but will fix many common issues such as a space after a comma, space at start of comment, blank lines among methods and classes, etc).

2. Improvements to the debugger, such as:
  • Thread creation is notified as threads are created instead of synchronized afterwards.
  • Support for using frame evaluation disabled by default as it made the debugger much slower on some cases.
  • Fixed case where breakpoint was missed if an exception was raised in a given line.
  • Properly break on unhandled exceptions on threads.
  • Add missing import which affected repl with IPython.
  • Fix for case where breakpoints could be missed.

As a note, the debugger improvements have been sponsored by Microsoft, which is in the process of using the PyDev Debugger as the core of ptvsd, the Python debugger package used by Python in Visual Studio and the Python Extension for Visual Studio Code (note that it's still marked as experimental there as it's in the process of being integrated into ptvsd).

 It's really nice to see pydevd being used in more IDEs in the Python world! 😉

Besides those, there are some bugfixes in handling f-strings and sending the contents of the current line to the console (through F2).

Also, a new major version of LiClipse (5.0) is now also available (see: http://www.liclipse.com/download.html for how to get it). It includes the latest PyDev and a new major Eclipse release (4.8 - Photon).

Saturday, May 12, 2018

Howto launch and debug in VSCode using the debug adapter protocol (part 2)

Ok, after the basic infracstructure, the next thing to do is actually launch some program without worrying about the debugger, so, we'll just run a program without being in debug mode to completion, show its output and terminate it when requested.

To launch a program, our debug adapter must treat the 'LaunchRequest' message and actually run the program (bear in mind that we'll just launch it without doing any debugging at this point).

The first point then is how to actually launch it. We provided options for the debugger to be launched with different console arguments specifying where to launch it (either just showing output to the debug console, using the integrated terminal or using an external terminal).

So, let's start with just showing output in the debug console.

Launching it should be simple: just generate a command line while treating the 'LaunchRequest' message, but then, based on the console specified some things may be different...

Let's start handling just showing the output on the debug console.

To do that we have launch the command properly redirecting the output to pipes (for python it's something as subprocess.Popen([sys.executable, '-u', file_to_run], stdout=subprocess.PIPE, stderr=subprocess.PIPE) and then create threads which will read that output to provide it back to vscode (so, when output is obtained, an OutputEvent must be given).

Also, create another thread so that when the process finishes, a TerminatedEvent is given (in python, just do a popen.wait() in a thread and when complete send the TerminatedEvent -- you may want to synchronize to make sure other threads related to output have finished before doing that).

At this point, we can run something and should be able to see anything printed both to stdout and stderr and when the process finishes, VSCode itself acknowledges that and closes the related controls.

Great! On to launching in the integrated terminal!

So, to launch in the terminal we have to first actually check if the client does support running in the terminal... in the InitializeRequest, if it is supported, we should've received in the arguments "supportsRunInTerminalRequest": True (if it doesn't, in my case I just fall back to the debug console).

This also becomes a little bit trickier because at this point we're the ones doing the request (RunInTerminalRequest) and the client should send a response (RunInTerminalResponse). So, on to it: when the client launches, create a RunInTerminalRequest with the proper kind ('internal' or 'external') and wait for the response.

At this point, the processId may not actually be available after launching in that mode (the RunInTerminalResponse processId is optional), which means that if we didn't really create a debugger (just a simple run), we're blind... we could do another program to launch it and return the pid to be able to notify that it was stopped and to kill it when needed, but this seems a bit overkill for me and I couldn't find any info on the proper behavior here, so, I decided that when the user chooses that mode with 'noDebug' I'll simply notify that the debug session is finished for the adapter with a TerminatedEvent (and the user can see the output and Ctrl+C it in the actual terminal).

As a note the 'noDebug' option is added behind the scenes by VSCode depending on whether the user has chosen to do a debug or run for the selected launch (so, it shouldn't be a part of the declared configuration in the extension).

Now, thinking a bit more about it, there's a caveat: when launching with the redirection to the debug console, we should treat sending to stdin too (we don't want to create a process he can't do any communication with later on).

To do that in 'noDebug' should be simple... when we receive an 'EvaluateRequest', we'll send it to stdin (when actually in debug mode we probably have to check the current debugger state to determine if we want to do an evaluation or send to stdin -- i.e.: if we are stopped in a breakpoint we may want to evaluate and not send to stdin).

As a note, after playing with it more I renamed the "console" option to "terminal" with options "none", "internal", "external" as I think that's a better representation of what's expected.

So, that's it for part 2: we're launching a program and redirecting the output as requested by the user (albeit without actually debugging it for now).