PyDev 5.4.0 is now available.
The main new feature in this release is support for Python 3.6 -- it's still not 100% complete, but at least all the new syntax is already supported (so, for instance, syntax and code analysis is already done, even inside f-strings, although code-completion is still not available inside them).
Also, it's now possible to launch modules using the python '-m' flag (so, PyDev will resolve the module name from the file and will launch it using 'python -m module.name'). Note that this must be enabled at 'Preferences > PyDev > Run'.
The last feature which I think is noteworthy is that the debugger can now show return values (note that this has a side-effect of making those variables live longer, so, if your program cares deeply about that, it's possible to disable it in Preferences > PyDev > Debug).
Now, for those that enjoy using PyDev, I've started a crowdfunding campaign at Patreon: https://www.patreon.com/fabioz so that PyDev users can help to keep it properly maintained... Really, without the PyDev supporters: https://www.brainwy.com/supporters/PyDev, it wouldn't be possible to keep it going -- thanks a lot! Hope to see you at https://www.patreon.com/fabioz too ;)
Wednesday, November 30, 2016
Friday, November 25, 2016
Python Import the world Anti-pattern!
I've been recently beaten from that anti-pattern which seems to be way too common in Python.
Now, what exactly is that "import the world" anti-pattern?
This anti-pattern (which I just made up) is characterized by importing lots of other files in the top-level of your scope of your module or package or doing too much at import time.
Now, you may ask: why is it bad? Every code I see in the Python world seems to be structured like that...
It's pretty simple actually: in Python, everything is dynamic, so, when you import a module you're actually making the Python interpreter load that file and run the bytecode, which will in turn generate classes, methods (and do anything which is in the global scope of your module or class definition)..
-- sure, even worse would be at import time going on to connect to some database or do other nasty stuff you wouldn't be expecting by just importing a module -- or who knows, going on to register to another service just because you imported some module! Importing code should be mostly free of side effects, besides, you know, generating those classes, methods and putting the module on sys.modules.
Ok, ok, I deviated from the main topic: why is it so bad having all those imports at the top-level of your module?
The reason is simple: nobody wants a gazillion of dependencies just because they partially imported some module for some simple operation.
It's slow, so much that any command line application that passed the toy stage -- and are concerned about the user experience -- have to hack around it.. really, just ask the mercurial guys.
It wastes memory (why loading all those modules if they won't be used anyways).
It adds dependencies which wouldn't be needed in the first place (like, you have a library which needs a lapack implementation which needs some ancient incarnations to be compiled that I don't care about because I won't be using the functions that need it in the first place).
It makes testing just a part of your code much slower (i.e.: you'll load 500 modules just for a small unit-test which touches just a small portion of your code).
It makes testing with pytest-xdist much slower (because it'll import all the code in all of its slaves instead of loading just what would be needed for a given worker).
So, please, just don't.
Ok, but does that really happen in practice?
Well, let me show you the examples I have stumbled in the last few days:
1. conda: Conda is a super-nice command line application to manage virtual environments and get dependencies. But let's take a look under the hood:
The main thing you use in conda is the command line, so, let's say you want to play with the "conda_env.cli.main" module. How long and how much does a simple: "from conda_env.cli import main"?
Let's see:
And it generates the following call graph:
Now, wait, what just happened? Haven't you just imported a module? Yes... I have, and in turn it has taken 0.3 seconds, loaded its configuration file under the hood (and made up some kind of global state?), parsed yaml, and imported lots of other things in turn (which I wish never happened) -- and it'd be even worse if you did a "conda env" command because it imports lots of stuff, parses the arguments and then decides to call a new command line with subprocess with "conda-env" and goes on to do everything again (see https://github.com/conda/conda/blob/85e52ebfe88c3e68f7cc5db699a8f4c450400c4b/conda/cli/conda_argparse.py#L150).
2. mock: Again, this is a pretty nice library (so much that it was added to the standard library on Python 3), but still, what do you expect from "import mock"?
Let's see:
And it generates the following call graph:
Ok, now there are less deps, but the time is roughly the same. Why? Because to define its version, instead of doing:
__version__ = '2.0.0'
it did:
And that went on to inspect lots of things system wide, including importing setuptools, which in turn parsed auxiliary files, etc... definitely not what I'd expect when importing some library which does mocks (really, setuptools is for setup time, not run time).
Now, how can this be solved?
Well, the most standard way is not putting the imports in the top-level scope. Just use local imports and try to keep your public API simple -- simple APIs are much better than complex APIs ;)
Some examples: if you have some library which wants to export some methods at some __init__ package, don't import the real implementations and put them in the namespace, just define the loads and dumps as methods in __init__ and make local imports which will do the real work loading the actual implementation as lazy imports inside those methods (or do import it, but then, make sure that the modules that contain the loads and dumps don't have global imports themselves).
Classes may be the more tricky case if you want them to be the bases for others to implement (because in this case you really need the class at the proper place for users of your API), so, here, you can explicitly import that class to be available in your __init__, but then, make sure that the scope which uses that class will import only what's needed to define that class, not to use it.
Please, don't try to use tricks such as https://github.com/bwesterb/py-demandimport for your own library... it just complicates the lives of everyone that wants to use it (and Python has the tools for you to work with that problem without having to resort to something which will change how Python imports behave globally) and don't try to load some global state behind the scenes (explicit is way better than implicit).
Maybe the ideal would be having Python itself do all imports lazily (but that's probably impossible right now) or have some support for a statement such as from xxx lazy_import yyy, so that you could just shove everything at your top-level, but until then, you can resort to using local imports -- note: you could still can create your own version of the lazy import which you could use to put things in the global scope, but as it's not standard, IDEs and refactoring tools may not always recognize it, so, I'd advise against it too given that local imports do work properly (although if you want to do some global registry of some kind, register as strings to be lazily loaded when needed instead of importing modules/classes to fill up your registry).
Really, this is not new: https://files.bemusement.org/talks/OSDC2008-FastPython ;)
Now, what exactly is that "import the world" anti-pattern?
This anti-pattern (which I just made up) is characterized by importing lots of other files in the top-level of your scope of your module or package or doing too much at import time.
Now, you may ask: why is it bad? Every code I see in the Python world seems to be structured like that...
It's pretty simple actually: in Python, everything is dynamic, so, when you import a module you're actually making the Python interpreter load that file and run the bytecode, which will in turn generate classes, methods (and do anything which is in the global scope of your module or class definition)..
-- sure, even worse would be at import time going on to connect to some database or do other nasty stuff you wouldn't be expecting by just importing a module -- or who knows, going on to register to another service just because you imported some module! Importing code should be mostly free of side effects, besides, you know, generating those classes, methods and putting the module on sys.modules.
Ok, ok, I deviated from the main topic: why is it so bad having all those imports at the top-level of your module?
The reason is simple: nobody wants a gazillion of dependencies just because they partially imported some module for some simple operation.
It's slow, so much that any command line application that passed the toy stage -- and are concerned about the user experience -- have to hack around it.. really, just ask the mercurial guys.
It wastes memory (why loading all those modules if they won't be used anyways).
It adds dependencies which wouldn't be needed in the first place (like, you have a library which needs a lapack implementation which needs some ancient incarnations to be compiled that I don't care about because I won't be using the functions that need it in the first place).
It makes testing just a part of your code much slower (i.e.: you'll load 500 modules just for a small unit-test which touches just a small portion of your code).
It makes testing with pytest-xdist much slower (because it'll import all the code in all of its slaves instead of loading just what would be needed for a given worker).
So, please, just don't.
Ok, but does that really happen in practice?
Well, let me show you the examples I have stumbled in the last few days:
1. conda: Conda is a super-nice command line application to manage virtual environments and get dependencies. But let's take a look under the hood:
The main thing you use in conda is the command line, so, let's say you want to play with the "conda_env.cli.main" module. How long and how much does a simple: "from conda_env.cli import main"?
Let's see:
>>> sys.path.append('C:\\Program Files\\Brainwy\\PyVmMonitor 1.0.1\\public_api')
>>> import pyvmmonitor
>>> pyvmmonitor.connect()
>>> len(sys.modules)
>>> 123
>>> @pyvmmonitor.profile_method
... def check():
... from conda_env.cli import main
...
>>> check()
>>> import sys
>>> len(sys.modules)
451
And it generates the following call graph:
Now, wait, what just happened? Haven't you just imported a module? Yes... I have, and in turn it has taken 0.3 seconds, loaded its configuration file under the hood (and made up some kind of global state?), parsed yaml, and imported lots of other things in turn (which I wish never happened) -- and it'd be even worse if you did a "conda env" command because it imports lots of stuff, parses the arguments and then decides to call a new command line with subprocess with "conda-env" and goes on to do everything again (see https://github.com/conda/conda/blob/85e52ebfe88c3e68f7cc5db699a8f4c450400c4b/conda/cli/conda_argparse.py#L150).
2. mock: Again, this is a pretty nice library (so much that it was added to the standard library on Python 3), but still, what do you expect from "import mock"?
Let's see:
>>> sys.path.append('C:\\Program Files\\Brainwy\\PyVmMonitor 1.0.1\\public_api')
>>> import pyvmmonitor
>>> pyvmmonitor.connect()
>>> len(sys.modules)
>>> 123
>>> @pyvmmonitor.profile_method
... def check():
... import mock
...
>>> check()
>>> import sys
>>> len(sys.modules)
291
And it generates the following call graph:
Ok, now there are less deps, but the time is roughly the same. Why? Because to define its version, instead of doing:
__version__ = '2.0.0'
it did:
from pbr.version import VersionInfo
_v = VersionInfo('mock').semantic_version()
__version__ = _v.release_string()
version_info = _v.version_tuple()
And that went on to inspect lots of things system wide, including importing setuptools, which in turn parsed auxiliary files, etc... definitely not what I'd expect when importing some library which does mocks (really, setuptools is for setup time, not run time).
Now, how can this be solved?
Well, the most standard way is not putting the imports in the top-level scope. Just use local imports and try to keep your public API simple -- simple APIs are much better than complex APIs ;)
Some examples: if you have some library which wants to export some methods at some __init__ package, don't import the real implementations and put them in the namespace, just define the loads and dumps as methods in __init__ and make local imports which will do the real work loading the actual implementation as lazy imports inside those methods (or do import it, but then, make sure that the modules that contain the loads and dumps don't have global imports themselves).
Classes may be the more tricky case if you want them to be the bases for others to implement (because in this case you really need the class at the proper place for users of your API), so, here, you can explicitly import that class to be available in your __init__, but then, make sure that the scope which uses that class will import only what's needed to define that class, not to use it.
Please, don't try to use tricks such as https://github.com/bwesterb/py-demandimport for your own library... it just complicates the lives of everyone that wants to use it (and Python has the tools for you to work with that problem without having to resort to something which will change how Python imports behave globally) and don't try to load some global state behind the scenes (explicit is way better than implicit).
Maybe the ideal would be having Python itself do all imports lazily (but that's probably impossible right now) or have some support for a statement such as from xxx lazy_import yyy, so that you could just shove everything at your top-level, but until then, you can resort to using local imports -- note: you could still can create your own version of the lazy import which you could use to put things in the global scope, but as it's not standard, IDEs and refactoring tools may not always recognize it, so, I'd advise against it too given that local imports do work properly (although if you want to do some global registry of some kind, register as strings to be lazily loaded when needed instead of importing modules/classes to fill up your registry).
Really, this is not new: https://files.bemusement.org/talks/OSDC2008-FastPython ;)
Thursday, November 03, 2016
PyDev 5.3.1 (bugfix release)
PyDev 5.3.1 is now out.
This is mostly a bugfix release.
It fixes a regression added on 5.3.0 where Ctrl+1 was not properly resolving symbols from dependent projects, code analysis on a particular case with PEP 448 (list unpacking), auto-indents async and await blocks, highlights async and await blocks and fixes an issue in the code completion using from..import..as aliases.
It does have one new feature in which Ctrl+Shift+O (which fixes unresolved tokens by adding the needed imports) uses the improved sorting also used for the code-completion (so that tokens from the project appear before tokens from other projects, which in turn appear before tokens from the standard library). Also, the substring completion is now on by default (it may be turned off in the code-completion preferences).
Enjoy ;)
This is mostly a bugfix release.
It fixes a regression added on 5.3.0 where Ctrl+1 was not properly resolving symbols from dependent projects, code analysis on a particular case with PEP 448 (list unpacking), auto-indents async and await blocks, highlights async and await blocks and fixes an issue in the code completion using from..import..as aliases.
It does have one new feature in which Ctrl+Shift+O (which fixes unresolved tokens by adding the needed imports) uses the improved sorting also used for the code-completion (so that tokens from the project appear before tokens from other projects, which in turn appear before tokens from the standard library). Also, the substring completion is now on by default (it may be turned off in the code-completion preferences).
Enjoy ;)
Friday, October 14, 2016
PyDev 5.3.0 (Code completion improvements, Validate code with Python 2 & 3 while typing, PyQt5 event loop in interactive console)
PyDev 5.3.0 is now available for download.
The major focus in this release was in the code completion, with a number of enhancements...
The main one is that it's now possible to request completions by matching a substring of the requested name, not only on the first chars (to make use of it, it has to be enabled in Preferences > PyDev > Editor > Code Completion > Match substrings on code completion).
Now, after this change was implemented, I noted that it was hard to use it by itself because as more completions are brought for a name it was hard to actually reach the one you wanted, so, it triggered to have even more improvements to the code completion, especially in the sorting, so, now completions in PyDev are re-sorted when typing so that based on the current name typed the matches that are exact matches or start with the requested token appear first, then as a second heuristic, the type of the completion is taken into account (i.e.: parameters appear before context-insensitive completions) and at last, where the completion was found plays a role, so that completions which are from the current project appear before referenced projects (which in turn appear before the standard library modules).
Also, the completions now take advantage of styling, so, the part of the completion which was matched is now shown in bold and the part of the completion which shows a qualifier for the completion is also properly styled.
All in all, I'm really enjoying using this new code completion mode, and will probably make it the default in the next release.
Besides this, the apply mode when holding Ctrl and Shift changed a bit... Ctrl on apply now always means only "replace current name with completion" and Shift (when focusing the completion pop-up) means "do local import" on completions with auto-import (it's important to note that the focus on the pop-up is important as Shift+Enter on the editor already has an important role as it cancels the completion and moves the cursor to a new line) .
Other niceties of this release include the possibility of specifying that a project should be checked with a Python 2 and a Python 3 grammar (customized in Project Properties > PyDev - Interpreter/Grammar > grammars for "additional syntax validation") and the interactive console now supports PyQt5 for interactively inspecting a widget while using the console (i.e.: it frees the event loop to process events in the interactive console as it already did for Gtk, PySide, PyQt4, etc). It may be activated with %matplotlib qt5 (when using IPython/matplotlib) or in Preferences > PyDev > Interactive Console > Enable GUI event loop integration > PyQt5.
Enjoy ;)
The major focus in this release was in the code completion, with a number of enhancements...
The main one is that it's now possible to request completions by matching a substring of the requested name, not only on the first chars (to make use of it, it has to be enabled in Preferences > PyDev > Editor > Code Completion > Match substrings on code completion).
Now, after this change was implemented, I noted that it was hard to use it by itself because as more completions are brought for a name it was hard to actually reach the one you wanted, so, it triggered to have even more improvements to the code completion, especially in the sorting, so, now completions in PyDev are re-sorted when typing so that based on the current name typed the matches that are exact matches or start with the requested token appear first, then as a second heuristic, the type of the completion is taken into account (i.e.: parameters appear before context-insensitive completions) and at last, where the completion was found plays a role, so that completions which are from the current project appear before referenced projects (which in turn appear before the standard library modules).
Also, the completions now take advantage of styling, so, the part of the completion which was matched is now shown in bold and the part of the completion which shows a qualifier for the completion is also properly styled.
All in all, I'm really enjoying using this new code completion mode, and will probably make it the default in the next release.
Besides this, the apply mode when holding Ctrl and Shift changed a bit... Ctrl on apply now always means only "replace current name with completion" and Shift (when focusing the completion pop-up) means "do local import" on completions with auto-import (it's important to note that the focus on the pop-up is important as Shift+Enter on the editor already has an important role as it cancels the completion and moves the cursor to a new line) .
Other niceties of this release include the possibility of specifying that a project should be checked with a Python 2 and a Python 3 grammar (customized in Project Properties > PyDev - Interpreter/Grammar > grammars for "additional syntax validation") and the interactive console now supports PyQt5 for interactively inspecting a widget while using the console (i.e.: it frees the event loop to process events in the interactive console as it already did for Gtk, PySide, PyQt4, etc). It may be activated with %matplotlib qt5 (when using IPython/matplotlib) or in Preferences > PyDev > Interactive Console > Enable GUI event loop integration > PyQt5.
Enjoy ;)
Thursday, August 18, 2016
PyDev 5.2.0 released (static type declarations, debugger fixes, IPython 5 support in interactive console)
PyDev 5.2.0 is now out (for LiClipse users, it's available in LiClipse 3.1.0).
This is the first release which provides support for PEP 484 (static type declarations), which means that if you are using a static type declaration on for a parameter or method return, PyDev should be able to provide you with code completion based on that information.
As a disclaimer, I personally don't use that feature -- although I resort to putting type information on docstrings sometimes -- Python for me is about duck-typing and being aconsenting adult responsible user, and type declarations don't seem to have a fit there for me... anyways, for those that think it's a good feature to have a more statically typed, Java-like structure / type system in Python, now PyDev is able to give you completions based on that information ;)
This release also has many other goodies that are noteworthy:
The debugger had a long standing issue where the variables wouldn't appear sometimes (and the user would have to re-select the stack frame for the variables to appear). In the latest release, PyDev should always show the variables properly (even if getting some variable takes time in the debugger backend). Also, when stepping over it should properly highlight in the variables view which variables changed and keep their expanded state.
The auto-indentation now better follows PEP 8, meaning that:
Note: it's possible to restore previous indent modes (which either always indented to the parenthesis level or always indented a single level) in the Preferences > PyDev > Editor > Typing.
Another good thing is that IPython 5 is now properly supported in the interactive console (IPython 5 did some backward-incompatible changes, so, PyDev code had to be adjusted accordingly). Also, some fixes were done in the interactive console to deal with multi-line statements in Jython and dealing with multiple statements in a single line.
Those were the major noteworthy things, but not all that was done. See: http://pydev.org for more information on what else was done in this release.
And as always, thank you very much to the PyDev supporters which help on keeping the PyDev development going strong.
This is the first release which provides support for PEP 484 (static type declarations), which means that if you are using a static type declaration on for a parameter or method return, PyDev should be able to provide you with code completion based on that information.
As a disclaimer, I personally don't use that feature -- although I resort to putting type information on docstrings sometimes -- Python for me is about duck-typing and being a
This release also has many other goodies that are noteworthy:
The debugger had a long standing issue where the variables wouldn't appear sometimes (and the user would have to re-select the stack frame for the variables to appear). In the latest release, PyDev should always show the variables properly (even if getting some variable takes time in the debugger backend). Also, when stepping over it should properly highlight in the variables view which variables changed and keep their expanded state.
The auto-indentation now better follows PEP 8, meaning that:
- Indenting directly after {, [, ( will add one indent level.
- Indenting after another token in a line with a {, [, ( will indent to the {, [, ( level.
Note: it's possible to restore previous indent modes (which either always indented to the parenthesis level or always indented a single level) in the Preferences > PyDev > Editor > Typing.
Another good thing is that IPython 5 is now properly supported in the interactive console (IPython 5 did some backward-incompatible changes, so, PyDev code had to be adjusted accordingly). Also, some fixes were done in the interactive console to deal with multi-line statements in Jython and dealing with multiple statements in a single line.
Those were the major noteworthy things, but not all that was done. See: http://pydev.org for more information on what else was done in this release.
And as always, thank you very much to the PyDev supporters which help on keeping the PyDev development going strong.
Friday, June 24, 2016
PyDev 5.1.2: pytest integration
PyDev 5.1.2 is now out. The major change is in the pytest integration.
For those that don't know about it, pytest (http://pytest.org) is a Python test framework which requires less scaffolding to make tests (you don't need a hierarchy such as in PyUnit, just test methods in a module and asserts for checks suffice, it takes care of providing a reasonable error message -- also, it has an interesting fixture concept which allows structuring the test environment in a way more natural then through an inheritance hierarchy with unittest.TestCase.setUp/tearDown).
If you want to use pytest, in PyDev, the runner in the preferences > PyDev > PyUnit has to be set as pytest.
It's interesting to note that PyDev makes it trivial to just run a single test: you can select the test by using the method navigation (Ctrl+Shift+Up and Ctrl+Shift+Down) and press Ctrl+F9: this will open the outline for selecting which tests to run with only that test selected by default, then you can press Enter to run the test (or Shift+Enter to debug it) -- note that this works with the regular unittest too, not only with pytest.
For those that don't know about it, pytest (http://pytest.org) is a Python test framework which requires less scaffolding to make tests (you don't need a hierarchy such as in PyUnit, just test methods in a module and asserts for checks suffice, it takes care of providing a reasonable error message -- also, it has an interesting fixture concept which allows structuring the test environment in a way more natural then through an inheritance hierarchy with unittest.TestCase.setUp/tearDown).
If you want to use pytest, in PyDev, the runner in the preferences > PyDev > PyUnit has to be set as pytest.
It's interesting to note that PyDev makes it trivial to just run a single test: you can select the test by using the method navigation (Ctrl+Shift+Up and Ctrl+Shift+Down) and press Ctrl+F9: this will open the outline for selecting which tests to run with only that test selected by default, then you can press Enter to run the test (or Shift+Enter to debug it) -- note that this works with the regular unittest too, not only with pytest.
Thursday, May 05, 2016
PyDev 5.0.0: PyUnit view persists state across restarts, Java 8 now required
PyDev 5.0.0 is now available for download.
The major feature in this release is that the PyUnit view will now persist its state across restarts ( personally, I find this pretty handy as I usually try to leave a failing test to be solved so that I know where I stopped ).
Also, this release requires Java 8 to be used ( if you need an older version, you can access the update site for older versions at: http://www.pydev.org/update_sites ).
As a note, if you're unsure on how to update your system java vm ( or it's not possible to ), you can dowload a .tar.gz version from http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html, extract it somewhere and point Eclipse to use it following the info from: http://wiki.eclipse.org/Eclipse.ini ( or just use LiClipse, which has the latest PyDev bundled for a hassle-free installation with many extra goodies ).
See: http://pydev.org for more details on this release ( as a note, PyDev 4.6.0 was previously released with the same contents, but was re-released under 5.0.0 to signal the major change on the required Java VM ).
The major feature in this release is that the PyUnit view will now persist its state across restarts ( personally, I find this pretty handy as I usually try to leave a failing test to be solved so that I know where I stopped ).
Also, this release requires Java 8 to be used ( if you need an older version, you can access the update site for older versions at: http://www.pydev.org/update_sites ).
As a note, if you're unsure on how to update your system java vm ( or it's not possible to ), you can dowload a .tar.gz version from http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html, extract it somewhere and point Eclipse to use it following the info from: http://wiki.eclipse.org/Eclipse.ini ( or just use LiClipse, which has the latest PyDev bundled for a hassle-free installation with many extra goodies ).
See: http://pydev.org for more details on this release ( as a note, PyDev 4.6.0 was previously released with the same contents, but was re-released under 5.0.0 to signal the major change on the required Java VM ).
Wednesday, March 23, 2016
PyDev 4.5.5: code-completion, pytest, python 3 parsing, debugger
PyDev 4.5.5 is now available for download.
This release improved code completion was improved so that namedtuple is recognized (although this requires that it's fields are passed in the constructor as a list of strings without being dynamically created) -- it should've supported super() too, but unfortunately there's still an issue in that feature (which should be solved in the next release).
This version also fixes some issues in the latest pytest integration (PyDev can use the pytest runner by enabling it in the preferences > PyDev > PyUnit and selecting pytest as the test runner), as a few things broke when used with the latest release due to some changes in pytest itself.
Another nice thing is that the parser for Python 3 now handles async and await in a backward-compatible way (as Python 3.5 does), which means it should be able to properly parse files from the Python 3.4 standard library again (which had some issues in the asyncio due to that) and fixes some of the unpacking generalizations from PEP 448 which weren't properly recognized.
Also, the UI now shows that the Python 3 parser is used from Python 3.0 to Python 3.5 (this is mostly because the adoption on Python 3 hasn't been so great so far, but now that it's becoming greater, the idea is that new versions will have a separate parser, so, when Python 3.6 is released, if it requires changes in the parser, a Python 3.6 parser will be created separate from the current parser).
And to finish, this release also fixes some issues in the debugger (the most critical related to an issue dealing with the connection to spawned processes in multiprocessing, which made debugging Celery fail under the debugger).
For LiClipse users, 2.5.4 was just released too with the latest PyDev!
This release improved code completion was improved so that namedtuple is recognized (although this requires that it's fields are passed in the constructor as a list of strings without being dynamically created) -- it should've supported super() too, but unfortunately there's still an issue in that feature (which should be solved in the next release).
This version also fixes some issues in the latest pytest integration (PyDev can use the pytest runner by enabling it in the preferences > PyDev > PyUnit and selecting pytest as the test runner), as a few things broke when used with the latest release due to some changes in pytest itself.
Another nice thing is that the parser for Python 3 now handles async and await in a backward-compatible way (as Python 3.5 does), which means it should be able to properly parse files from the Python 3.4 standard library again (which had some issues in the asyncio due to that) and fixes some of the unpacking generalizations from PEP 448 which weren't properly recognized.
Also, the UI now shows that the Python 3 parser is used from Python 3.0 to Python 3.5 (this is mostly because the adoption on Python 3 hasn't been so great so far, but now that it's becoming greater, the idea is that new versions will have a separate parser, so, when Python 3.6 is released, if it requires changes in the parser, a Python 3.6 parser will be created separate from the current parser).
And to finish, this release also fixes some issues in the debugger (the most critical related to an issue dealing with the connection to spawned processes in multiprocessing, which made debugging Celery fail under the debugger).
For LiClipse users, 2.5.4 was just released too with the latest PyDev!
Friday, January 22, 2016
PyDev 4.5.3: Debugger fixes and improvements in PyUnit view
PyDev 4.5.3 is now available for download.
This release was done in such a short notice from the last release because a critical issue was fixed in the debugger (although it also added a few improvements to the PyUnit view, making the history dropdown show more information on the run and providing a way to export the results of a run to the clipboard, which are pretty nice).
As a note, LiClipse users (especially those using dark themes) should benefit a lot from the latest release, as it themes the trees and tables scrollbars (with the support from https://github.com/fappel/xiliary), so, it's a highly recommended download ;)
Anyways, I'll take the chance to talk about the bug which actually triggered this release (which was https://sw-brainwy.rhcloud.com/tracker/PyDev/650):
The PyDev debugger by default will monkey-patch all the calls which create a new process to automatically connect the debugger to the spawned processes if those are also Python processes -- and it also monkey-patches the os.fork to connect the forked process to the debugger.
Now, in os.fork it always connected the new process to the debugger, as it expected the new process to be a Python process, but there's a catch here: on Linux, subprocess.Popen will first fork the current process and then will do an os.execvp to replace the forked process with a new process, but in the latest version this was making the debugger crash (although it's still not 100% clear to me why that same process wasn't crashing before as the debugger already did this in previous versions -- anyways, os.execvp did some incantation under the hood which crashed when the debugger was setup when it was doing something as subprocess.Popen(['uname', '-m']), which first forked the process, connected it to the debugger and then replaced it with the uname executable).
So, the actual fix was detecting that it was forking for a subprocess.Popen and refrain from setting up the debugger if it was not executing a new Python program ;)
Enjoy!
This release was done in such a short notice from the last release because a critical issue was fixed in the debugger (although it also added a few improvements to the PyUnit view, making the history dropdown show more information on the run and providing a way to export the results of a run to the clipboard, which are pretty nice).
As a note, LiClipse users (especially those using dark themes) should benefit a lot from the latest release, as it themes the trees and tables scrollbars (with the support from https://github.com/fappel/xiliary), so, it's a highly recommended download ;)
Anyways, I'll take the chance to talk about the bug which actually triggered this release (which was https://sw-brainwy.rhcloud.com/tracker/PyDev/650):
The PyDev debugger by default will monkey-patch all the calls which create a new process to automatically connect the debugger to the spawned processes if those are also Python processes -- and it also monkey-patches the os.fork to connect the forked process to the debugger.
Now, in os.fork it always connected the new process to the debugger, as it expected the new process to be a Python process, but there's a catch here: on Linux, subprocess.Popen will first fork the current process and then will do an os.execvp to replace the forked process with a new process, but in the latest version this was making the debugger crash (although it's still not 100% clear to me why that same process wasn't crashing before as the debugger already did this in previous versions -- anyways, os.execvp did some incantation under the hood which crashed when the debugger was setup when it was doing something as subprocess.Popen(['uname', '-m']), which first forked the process, connected it to the debugger and then replaced it with the uname executable).
So, the actual fix was detecting that it was forking for a subprocess.Popen and refrain from setting up the debugger if it was not executing a new Python program ;)
Enjoy!
Thursday, January 14, 2016
PyDev 4.5.1: debug faster
PyDev 4.5.1 brings niceties such as an improvement on the code-completion so that properties declared with @property no longer show arguments parenthesis and Ctrl+1 can be used inside a bracketed statement to wrap/unwrap its contents over multiple lines (thanks to yohell). Also, the refactoring, parsing and search had some fixes, but the major changes were in the debugger.
The debugger is much faster and has optional Cython modules for even additional speedups.
In short, the debugger is overall 40% faster without Cython and 138% faster with Cython in the benchmarks created -- although in real world cases I expect even better gains as the benchmarks were done for the cases where the debugger has more overhead than usual.
Graphs with more details on the improvements may be seen at:
https://www.speedtin.com/reports/7_pydevd_cython (performance results with cython).
https://www.speedtin.com/reports/8_pydevd_pure_python (performance results without cython).
Also, the debugger backend is now also available through "pip install pydevd" (https://pypi.python.org/pypi/pydevd), so, it's easier to setup the remote debugging in a different machine (note that users shouldn't need to install the debugger, only if doing a remote debugging session).
As a note, I'd like to thank JetBrains, which helped on sponsoring the performance improvements in the PyDev Debugger (as it's also the debugger backend used by PyCharm).
For LiClipse users, 2.5.1 (just released too) includes the latest PyDev.
The debugger is much faster and has optional Cython modules for even additional speedups.
In short, the debugger is overall 40% faster without Cython and 138% faster with Cython in the benchmarks created -- although in real world cases I expect even better gains as the benchmarks were done for the cases where the debugger has more overhead than usual.
Graphs with more details on the improvements may be seen at:
https://www.speedtin.com/reports/7_pydevd_cython (performance results with cython).
https://www.speedtin.com/reports/8_pydevd_pure_python (performance results without cython).
Also, the debugger backend is now also available through "pip install pydevd" (https://pypi.python.org/pypi/pydevd), so, it's easier to setup the remote debugging in a different machine (note that users shouldn't need to install the debugger, only if doing a remote debugging session).
As a note, I'd like to thank JetBrains, which helped on sponsoring the performance improvements in the PyDev Debugger (as it's also the debugger backend used by PyCharm).
For LiClipse users, 2.5.1 (just released too) includes the latest PyDev.
Subscribe to:
Posts (Atom)