Thursday, August 11, 2022

PyDev debugger: Going from async to sync to async... oh, wait.

 In Python asyncio land it's always a bit of a hassle when you have existing code which runs in sync mode which needs to be retrofitted to run in async, but it's usually doable -- in many cases, slapping async on the top of a bunch of definitions and adding the needed await statements where needed does the trick -- even though it's not always that easy.

Now, unfortunately a debugger has no such option. You see, a debugger needs to work on the boundaries of callbacks which are called from python (i.e.: it will usually do a busy wait from a line event from a callback registered in sys.settrace which is always called as a sync call).

Still, users still want to do some evaluation in the breakpoint context which would await... What now? Classic questions of how to go from async to sync say this is not possible.

This happens because to run something in asynchronous fashion an asyncio loop must be used to run it, but alas, the current loop is paused in the breakpoint and due to how asyncio is implemented in Python the asyncio loop is not reentrant, so, we can't just ask the loop to keep on processing at a certain point -- note that not all loops are equal, so, this is mostly an implementation detail on how CPython has implemented it, but unless we want to monkey-patch many things to make it reentrant, this would be a no-no... also, even if possible, it's not possible in asyncio to force a given coroutine to execute, rather we schedule it and asyncio decides when it'll run afterwards).

My initial naive attempt was just creating a new event loop, but again, CPython gets in the way because 2 event loops can't even coexist in the same thread. Then I thought about recreating the asyncio loop and got a bit further (up to being able to evaluate an asyncio.sleep coroutine), but after checking the asyncio AbstractEventLoop it became clear that the API is just too big to reimplement safely (it's not just about implementing the loop, it's also about implementing network I/O such as getnameinfo, create_connection, etc).

In the end the solution implemented for the debugger is that to support await constructs for evaluation, a new thread is created with a new event loop and that event loop in that new thread will execute the coroutine (with the context of the paused frame passed to that thread for the evaluation).

This is not perfect as there are some cons, for instance, evaluating the code in a thread can mean that some evaluations may not work because some frameworks such as qt consider the UI thread as special and won't work properly, checks for the current thread won't match the thread paused and probably a bunch of other things, but I guess it's a reasonable tradeoff vs not having it at all as it should work in the majority of cases.

Keep an eye open for the next release as it'll be possible to await coroutines in the debugger evaluation and watches ;)

p.s.: For VSCode users this will also be available in debugpy.

Thursday, March 10, 2022

PyDev 9.3.0 (debugger improvements / last version with Python 2.7 - 3.5 support)

PyDev 9.3.0 is now available.

The main changes in this release are related to the debugger, with improvements such as:

  • Major issue fixed issue where variable children sometimes wouldn't expand correctly.
  • Fixed some case where automatic connection to subprocesses wouldn't work.
  • Debugging with Pandas is much improved with the addition of some custom converters.
  • Opt-in support to show paused greenlets by setting GEVENT_SHOW_PAUSED_GREENLETS=1.
  • Support for newer versions of gevent.
  • Information on user settings paths is cached to fix issue with slow debugging using pipenv project.
  • A warning is shown if getting some attribute / getting its repr is slow.
  • Interactively inspect matplotlib plots when the QtAgg backend is used.
  • Support for PySide2.
  • Better error messages in case Python 3.11 frozen modules are being used.

Also noteworthy is that this will be the last release supporting older versions of Python including Python 2.7 up to Python 3.5. Newer releases will only support Python 3.6 onwards.

Sunday, April 18, 2021

PyDev 8.3.0 (Java 11, Flake 8 , Code-completion LRU, issue on Eclipse 4.19)

PyDev 8.3.0 is now available!

Let me start with some warnings here:

First, PyDev now requires Java 11. I believe that Java 11 is pretty standard nowadays and the latest Eclipse also requires Java 11 (if you absolutely need Java 8, please keep using PyDev 8.2.0 -- or earlier -- indefinitely, otherwise, if you are still using Java 8, please upgrade to Java 11 -- or higher).

Second, Eclipse 2021-03 (4.19) is broken and cannot be used with any version of PyDev due to https://bugs.eclipse.org/bugs/show_bug.cgi?id=571990, so, if you use PyDev, please keep to Eclipse 4.18 (or get a newer if available) -- the latest version of PyDev warns about this, older versions will not complain but some features will not function properly, so, please skip on using Eclipse 4.19 if you use PyDev.

Now, on to the goodies ;)

On the linters front, the configurations for the linters can now be saved to the project or user settings and flake8 has an UI for configuration which is much more flexible, allowing to change the severity of any error.

A new option which allows all comments to be added to a single indent was added (and this is now the default).

The code-completion and quick fixes which rely on automatically adding some import will now cache the selection so that if a given token is imported that selection is saved and when asked again it'll be reused (so, for instance, if you just resolved Optional to be typing.Optional, that'll be the first choice the next time around).

Environment variables are now properly supported in .pydevproject. The expected format is: ${env_var:VAR_NAME}.

Acknowledgements

Thanks to Luis Cabral, who is now helping in the project for doing many of those improvements (and to the Patrons at https://www.patreon.com/fabioz which enabled it to happen).

 Enjoy!

Saturday, February 27, 2021

PyDev 8.2.0 released (external linters, Flake8, path mappings, ...)

 PyDev 8.2.0 is now available for download.

This release has many improvements for dealing with external linters.

The main ones are the inclusion of support for the Flake8 linter as well as using a single linter call for analyzing a directory, so, that should be much faster now (previously it called external linters once for each file) .

Note: to request code analysis for all the contents below a folder, right-click it and choose PyDev > Code analysis:

Another change is that comments are now added to the line indentation... 

This means that some code as:

  def method():  
     if True:  
         pass

Will become:

  def method():  
     # if True:  
         # pass

p.s.: it's possible to revert to the old behavior by changing the preferences at PyDev > Editor > Code Style > Comments. 

Also note that after some feedback, on the next release an option to format such as the code below will also be added (and will probably be made the default):

  def method():  
     # if True:  
     #     pass

Interpreter configuration also got a revamp:

So, it's possible to set a given interpreter to be the default one and if you work with conda, select Choose from Conda to select one of your conda environments and configure it in PyDev.

Path mappings for remote debugging can now (finally) be configured from within PyDev itself, so, changing environment variables is no longer needed for that:

 

Note that Add path mappings template entry may be clicked multiple times to add multiple entries.

That's it... More details may be found at: http://pydev.org.

Hope you enjoy the release 😊


Tuesday, December 08, 2020

PyDev 8.1.0 released (Python 3.9, Code analysis, f-string quick-fixes)

 PyDev 8.1.0 is now available for download.

As a note, I didn't really create a post on 8.0.1, so, I'm covering some of the features in that version in this blog post too! 😊

Some nice things added: 

  • Python 3.9 is officially supported in PyDev -- definitely a must if you plan on using Python 3.9!
  • There are quick-fixes (Ctrl+1) to convert a string into an f-string (see image below):


Note that even if there's no formatting in the string, with this it's possible to write some string as: "Value is {value}" and then use the quick fix just to add the "f" to the front of the string (which is quite handy if you weren't initially planning to create it as an f-string).

  • When starting the interactive console, it's now possible to save how the interactive console should be initialized (see image below):

Note that it's also possible to change those settings in the interactive console preferences if you want to change them after the initial selection (hint: after the interactive console is created it's possible to use F2 to send a line from the editor for execution to the interactive console for notebook-like evaluation).

Besides this there are many other improvements, such as improved conda activation, rerunning pytest parametrized tests from the PyUnit view, MyPy integration improvements, support for from __future__ import annotations in code analysis and debugger improvements, among many others.. see: https://pydev.org for more details.

Enjoy!