Thursday, June 11, 2020

PyDev 7.6.0 (Python 3.8 parsing fixes and debugger improvements)

PyDev 7.6.0 is now available for download.

This release brings multiple fixes to parsing the Python 3.8 grammar (in particular, dealing with f-strings and iterable unpacking had some corner cases that weren't well supported).

Also, the debugger had a number of improvements, such as:
  • Grouping variables which were previously hidden -- note that they can be hidden/shown when debugging from the variables view dropdown menu as shown in the screen below:

  • Better support for PySide2;
  • Show the disassembly of frames when sources aren't available:



Besides those, there were a number of other improvements... some noteworthy are support for the latest PyTest, faster PyDev Package Explorer, type inference for type comments with self attributes (i.e.: #: :type self.var: MyClass) and properly recognizing trailing commas on automatic import.

Acknowledgements

Thanks to Luis Cabral, who is now helping in the project for doing many of those improvements (and to the Patrons at https://www.patreon.com/fabioz which enabled it to happen).

Thanks for Microsoft for sponsoring the debugger improvements, which are also available in Python in Visual Studio and the Python Extension for Visual Studio Code.

Enjoy!
--
Fabio

Wednesday, March 18, 2020

How is frame evaluation used in pydevd?

First some background in frame evaluation:

Since Python 3.6, CPython has a mechanism which allows clients to override how it evaluates frames. This is done by changing PyThreadState.interp.eval_frame to a different C-function (the default being _PyEval_EvalFrameDefault). See: pydevd_frame_evaluator.pyx#L370 in pydevd (note that Cython is used there).

Note that this affects the Python runtime globally, whereas the regular Python tracing function -- set through sys.settrace() -- affects only the current thread (so, some of the caches for frame evaluation in pydevd are thread-local due to that).

How is this used in the debugger?

Well, the debugger doesn't really want to change how Python code is executed, but, there's another interesting side effect of the frame evaluation: it's possible to change the bytecode of the frame right before it's evaluated and CPython will interpret that bytecode instead of the original bytecode of the frame.

So, this works the following way: the frame evaluation function receives a PyFrameObject*, and at that point, the debugger checks the frame for existing breakpoints, if it has a breakpoint, it'll create a new code object which has a programmatic breakpoint (pydevd_frame_evaluator.pyx#L234) and change PyFrameObject.f_code to point to the new code object (pydevd_frame_evaluator.pyx#L358) -- when it reaches the programmatic breakpoint (pydevd_frame_tracing.py#L34), the regular (trace-based) debugger will kick in at that frame. Until that breakpoint is reached, frames are executed at full speed.

But if it runs at full speed, why is my program still running slower when using pydevd with frame evaluation?

Well, frames are executed at full speed, but, the debugger still adds some overhead at function calls (when it decides whether to add the programmatic breakpoint) and it also needs to add an almost no-op trace (pydevd_frame_evaluator.pyx#L95) function to sys.settrace -- which makes function calls slower too (this is needed because otherwise the debugger is not able to switch to the regular tracing by just changing the frame.f_trace as frame.f_trace is only checked when a tracing function is set for some thread through sys.settrace()). There are also some cases where it can't completely skip tracing for a frame even if it doesn't have a breakpoint (for instance, when it needs to break on caught exceptions or if it's stepping in the debugger).

It's interesting to note that even the regular (tracing) debugger on pydevd can run frames at full speed (it evaluates all frames and if a frame doesn't have a breakpoint the tracing for that frame will be skipped), the difference is that if a frame does have a breakpoint, that frame can run at full speed until it reaches the breakpoint in the frame eval mode, whereas in the regular mode each new line tracing event would need to be individually checked for a breakpoint.

If it just changes the bytecode, why use frame eval at all, can't you just change the bytecode of objects at a custom import hook? (which could have the benefit of avoiding the performance penalty of checking the frame on each new frame call)

There are 2 main reasons for that: the 1st is that breakpoints can change and when they change the frame evaluation would need to be completely shut down and only the tracing debugger would be valid from that point onwards (whereas right now, if breakpoints change, the tracing debugger kicks in for all the frames that were currently running but the frame evaluation debugger can still be used for new frames). The 2nd is that it can be hard to consistently do that if not just before frame evaluation (because user code can also change a method code and there are a number of corner cases to change the bytecode for live objects -- think of a function inside a function or decorated functions).

Note that this means that the debugger could probably get away with something simpler than frame evaluation and could potentially be applicable to other Python implementations (say, a different callback just before the frame is evaluated which allows to change the frame code... unfortunately it can't currently be done through the "call" event received by the trace function set by sys.settrace because at that point the frame is already being evaluated with the current code and at that point, even if it's changed, Python won't pick up that change).

That's it, hope you enjoyed pydevd using frame evaluation for debugging purposes 101 ;)





Friday, January 10, 2020

PyDev 7.5.0 Released (Python 3.8 and Cython)

PyDev 7.5.0 is now available for download.

The major changes in this release are Python 3.8 support and improved Cython parsing.

Python 3.8 should've been in 7.4.0 (but because of an oversight on my part during the build it wasn't, so, this release fixes that).

As for the Cyhon AST, Cython is now parsed using Cython itself (so, it needs to be installed and available in the default interpreter for PyDev to be able to parse it). The major issue right now is that the parser is not fault tolerant (this means that for code-completion and code-analysis to kick in the code needs to be syntax-correct, which is a problem when completing for instance variables right after a dot).

Fixing that in Cython seems to be trivial (https://github.com/cython/cython/issues/3303), but I'm still waiting for a signal that it's ok to add that support to make Cython parsing fault-tolerant.

Enjoy!

Wednesday, March 27, 2019

PyDev 7.2.0 released

PyDev 7.2.0 is now available for download.

This version brings some improvements to the debugger and a fix for when PyDev could not properly find pipenv which could impact some users.

See: http://pydev.org for more details.

Friday, November 09, 2018

PyDev 7.0 (mypy, black, pipenv, faster debugger)

PyDev 7.0 (actually, PyDev 7.0.3 after some critical bugfixes on 7.0.0) is now available.

Some of the improvements available in this version include:

Mypy may be used as an additional backend for code analysis (see the preferences in the Preferences > PyDev > Editor > Code Analysis > Mypy). It is is similar to the PyLint integration, and will run Mypy whenever a file is saved in the editor.

Black can be used as the code-formatting engine (so, it's now possible to choose between the PyDev formatter, autopep8 or Black).

pipenv may be used for managing virtual environments (so, when creating a new project, clicking to configure an interpreter not listed will present an option to create a new interpreter with pipenv).

Improvements managing interpreters: it's now possible to manage interpreters directly from the editor (so, for instance, doing ctrl+2 pip install django will use pip to install django in the interpreter related to the opened editor -- and it's possible to change pip for conda or pipenv too).

Debugger improvements

The debugger is much faster for Python 3.6 onwards (when cython compiled extensions are available).

The performance improvement on the debugger is due to the using the frame eval mode for breakpoints again.

To give some history, that mode was previously disabled on pydevd because it had some issues which made function calls much slower (even though line stepping was zero overhead), but in the end, that overhead could make the performance worse than just using the plain tracing mode. Also, it had a big memory leak and in the cases where the tracing mode had to be reenabled both modes would be active at the same time and performance would suffer quite a bit.

-- all of those should be fixed and it should now perform better or at least on par with the tracing mode with cython in all scenarios... if you like numbers, see: https://github.com/fabioz/PyDev.Debugger/blob/master/tests_python/performance_check.py#L193 for some benchmarks.

p.s.: the improvements in the Debugger were sponsored by Microsoft, as the PyDev Debugger is used as the core of ptvsd, the Python debugger package used by Python in Visual Studio and the Python Extension for Visual Studio Code.

 

Monday, September 03, 2018

PyDev 6.5.0 (#region code folding)

PyDev 6.5.0 is now available for download.

There are some nice features and fixes available in this release:
  • #region / #endregion comments can now be used by the code-folding engine.
  • An action to easily switch the default interpreter is now available (default binding: Ctrl+Shift+Alt+I -- note that it must be executed with an opened editor).
  • It's possible to create local imports from global imports (use Ctrl+1 on the name of a given global import and select "Move import to local scope(s)" -- although note that the global import needs to be manually deleted later on).
  • The interactive interpreter now has scroll-lock.
  • The debugger is much more responsive!
See: http://www.pydev.org for more details.

Monday, August 13, 2018

Profiling pytest startup

I'm a fan of pytest (http://www.pytest.org), yet, it seems that the startup time for running tests locally in the app I'm working on is slowly ramping up, so, I decided to do a profile to see if there was anything I could do to improve that.

The first thing I did was creating a simple test and launching it from the PyDev (http://www.pydev.org/) profile view -- it enables any launch done in PyDev to show its performance profile on PyVmMonitor (https://www.pyvmmonitor.com).

Note that this is an integration test that is starting up a big application, so, the total time just to startup all the fixtures which make the application live and shut down the fixtures is 15 seconds (quite a lot IMHO).

The first thing I noticed looking the profile is that 14% of that time seems to be creating a session temp dir:


After investigating a bit more it seems that there is a problem in the way the fixture used make_numbered_dir (it was passing a unicode when it should be a str on Python 2) and make_numbered_dir had an issue where big paths were not removed.

So, pytest always visited my old files every time I launched any test and that accounted for 1-2 seconds (I reported this particular error in: https://github.com/pytest-dev/pytest/issues/3810).

Ok, down from 15 to 13 seconds after manually removing old files with big paths and using the proper API with str on Py2.

Now, doing a new profile with that change has shown another pytest-related slowdown doing rewrites of test cases. 


This is because of a feature of pytest where it'll rewrite test files to provide a prettier stack trace when there's some assertion failure.

So, I passed --assert=plain to pytest and got 3 more seconds (from 13 down to 10) -- it seems all imports are a bit faster with the import rewrite disabled, so, I got an overall improvement there, not only in that specific part of the code (probably not nice on CI where I want to have more info, but seems like a nice plus locally, where I run many tests manually as I think the saved time for those runs will definitely be worth it even with less info when some assertion fails).

Now, with that disabled the next culprit seems to be getting its plugins to load:



But alas, it uses setuptools and I know from previous experience that it's very hard to improve that (it is very greedy in the way it handles loading metadata, so, stay away unless you're ok in wasting a lot of time on your imports) and the remainder of the time seems to be spread out importing many modules -- the app already tries to load things as lazy as possible... I think I'll be able to improve on that to delay some imports, but Python libraries are really hard to fix as everyone imports everything in the top of the module.

Well, I guess going from 15 s to 10 s with just a few changes is already an improvement in my case for an integrated tests which starts up the whole app (although it could certainly be better...) and I think I'll still be able to trim some of that time doing some more imports lazily -- although that's no longer really pytest-related, so, that's it for this post ;)