Creating extension to profile Python with PyVmMonitor from Visual Studio Code

Ok, so, the target here is doing a simple extension with Visual Studio Code which will help in profiling the current module using PyVmMonitor (http://www.pyvmmonitor.com/).

The extension will provide a command which should open a few options for the user on how he wants to do the profile (with yappi, cProfile or start without profiling but connected with the live sampling view).

I went with https://code.visualstudio.com/docs/extensions/yocode to bootstrap the extension, which gives a template with a command to run then renamed a bunch of things (such as the extension name, description, command name, etc).

Next was finding a way to ask the user for the options (to ask how the profile should be started). Searching for it revealed https://tstringer.github.io/nodejs/javascript/vscode/2015/12/14/input-and-output-vscode-ext.html, so, I went with creating the needed constants and going with vscode.window.showQuickPick (experimenting, undefined is returned if the user cancel the action, so, that needs to be taken into account too).

Now, after the user chooses how to start PyVmMonitor, the idea would be making any launch actually start in the chosen profile mode (which is how it works in PyDev).

After investigating a bit, I couldn't find out how to intercept an existing launch to modify the command line to add the needed parameters for profiling with PyVmMonitor, so, this integration will be a bit more limited than the one in PyDev as it will simply create a new terminal and call PyVmMonitor asking it to profile the currently opened module...

In the other integrations, it was done as a setting where the user selected that it wanted to profile any python launch from a given point onward as a toggle and then intercepted launches changing the command line given, so, for instance, it could intercept a unittest launch too, but in this case, it seems that there's currently no way to do that -- or some ineptitude on my part finding an actual API to do it ;)

Now, searching on the VSCode Python plugin, I found a "function execInTerminal", so, I based the launching in it (but not using its settings as I don't want to add a dependency on it for now, so, I just call `python` -- if that's wrong, as it opens a shell, the user is free to cancel that and correct the command line to use the appropriate python interpreter or change it as needed later on).

Ok, wrapping up: put the initial version of the code on https://github.com/fabioz/vscode-pyvmmonitor. Following https://code.visualstudio.com/docs/extensions/publish-extension did work out, so, there's a "Profile Python with PyVmMonitor" extension now ;).

Some notes I took during the process related to things I stumbled on or found awkard:

• After publishing the first time and installing, the extension wasn't working because I wrongly put a dependency from npm in "devDependencies" and not in "dependencies" (the console in the developer tools helped in finding out that the dependency wasn't being loaded after the extension was installed).
• When a dependency is added/removed, npm install needs to be called again, it's not automatic.
• When uploading the extension I had the (common) error of not generating a token for "all" ;)
• Apparently there's a "String" and a "string" in TypeScript (or at least within the dependencies when editing VSCode).
• The whole launching on VSCode seems a bit limited/ad-hoc right now (for instance, .launch files create actual launchers which can be debugged but the python extension completely bypasses that by doing a launch in terminal for the current file -- probably because it's a bit of a pain creating that .launch file) -- I guess this reflects how young VSCode is... on the other hand, it really seems it built upon previous experience as the commands and bindings seems to have evolved directly to a good design (Eclipse painfully iterated over several designs on its command API).
• Extensions seem to be limited by design. I guess this is good and bad at the same time... good that extensions should never slow down the editor, but bad because they are not able to do something which is not in the platform itself to start with -- for instance, I really missed a good unittest UI when using it... there are actually many other things I missed from PyDev, although I guess this is probably a reflect on how young the platform is (and it does seem to be shaping up fast).

PyDev 6.2.0: Interactive Console word wrapping, pytest hyperlinking

PyDev 6.2.0 is mostly a bugfix release, although it does bring some features to the table to such as adding the possibility of activating word-wrapping in the console and support for code-completion using the Python 3.6 variable typing.

Another interesting change is that pytest filenames are properly hyperlinked in the console (until now PyDev resorted to mocking some functions of pytest so that when it printed exceptions it used the default Python traceback format -- now that's no longer done).

See: http://www.pydev.org for complete details.

p.s.: Thank you to all PyDev supporters -- https://www.brainwy.com/supporters/PyDev-- which enable PyDev to keep on being improved!

p.s.: LiClipse 4.4.0 already bundles PyDev 6.2.0, see: http://www.liclipse.com/download.html for download links.

PyDev 6.1.0: dealing with blank lines in code formatter

PyDev 6.1.0 is now available for download. The major change in this release is in the code formatter, which can now deal with adding or removing blank lines so that code can properly conform to pep-8, besides having a number bugs fixed (see http:///www.pydev.org for more details).

Now, why use the PyDev code formatter at all when there are so many other options available? (i.e.: autopep8, yapf, PythonTidy -- and autopep8 is even already included by default in PyDev) 

Well, the PyDev code formatter is unique in that it tries to do as few changes as possible to the code, so, it tries to conform to the coding format that the programmer uses, just fixing few (usually obvious) issues, such as spaces after comma, spaces in comments, operators or right-trimming lines, with an option to actually only fix only the lines actually changed.

-- for actually changing the indentation of statements or comments, PyDev has options which can be manually activated, such as wrap or unwrap statement -- through Ctrl+1, wrap statement or Ctrl+1, unwrap statement in the line which has the contents to be wrapper or unwrapped or Ctrl+2, w to wrap comments -- see: http://pydev.blogspot.com.br/2015/04/wrapping-docstringscomments-in-pydev.html.

Also, the PyDev code formatter is pretty fast, so, I don't have issues in letting the option to autoformat on save turned on (speed is the main reason why I added such a feature to the PyDev code formatter instead of going with autopep8 or integrating another code formatting tool).

So, that's it, enjoy!

p.s.: Thank you to all PyDev supporters -- https://www.brainwy.com/supporters/PyDev/ -- which enable PyDev to keep on being improved!

p.s.: LiClipse 4.3.1 already bundles PyDev 6.1.0, see: http://www.liclipse.com/download.html for download links.

PyDev 6.0: pip, conda, isort and subword navigation

The new PyDev release is now out and offers some really nice features on a number of fronts!

The interpreter configuration now integrates with both pip and conda, showing the installed packages and allowing any package to be installed and uninstalled from inside the IDE.

Also, it goes a step further in the conda integration and allows users to load the proper environment variables from the env -- this is actually false by default and can be turned on in the interpreter configuration page when PyDev identifies an interpreter as being managed by conda by checking the "Load conda env vars before run" configuration (so, if you have some library which relies on some configuration you don't have to activate the env outside the IDE).

Another change which is pretty nice is that now when creating a project there's an option to specify that the project should always use the interpreter version for syntax validation.

Previously a default version for the grammar was set, but users could be confused when the version didn't match the interpreter... note that it's still possible to set a different version or even add additional syntax validators, for cases when you're actually dealing with supporting more than one Python version.

The editor now has support for subword navigation (so, navigating words as MyReallyNiceClass with Ctrl+Left/Right will stop after each subword -- i.e.: 'My', 'Really', 'Nice', 'Class' -- remember that Shift+Alt+Up can be used to select the full word for the cases where Ctrl+ShiftLeft/Right did it previously).

This mode is now also consistent among all platforms (previously each platform had its own style based on the underlying platform -- it's still possible to revert to that mode in the Preferences > PyDev > Editor > Word navigation option).

Integration with PyLint and isort were also improved: the PyLint integration now provides an option to search for PyLint in the interpreter which a project is using and isort integration was improved to know about the available packages (i.e.: based on the project/interpreter configuration, PyDev knows a lot about which should be third party/ library projects and passes that information along to isort).

In the unittest front, Robert Gomulka did some nice work and now the name of the unittest being run is now properly shown in the run configuration and it's possible to right-click a given selection in the dialog to run tests (Ctrl+F9) and edit the run configuration (to edit environment variables, etc) before running it.

Aside from that there were also a number of other fixes and adjustments (see http://pydev.org for more details).

Enjoy!

p.s.: Thank you to all PyDev supporters -- https://www.brainwy.com/supporters/PyDev/ -- which enable PyDev to keep on being improved!

p.s.: LiClipse 4.2.0 already bundles PyDev 6.0, see: http://www.liclipse.com/download.html for download links.

PyDev 5.9.2 released (Debugger improvements, isort, certificate)

PyDev 5.9.2 is now available for download.

This version now integrates the performance improvements which were done in PyDev.Debugger for 3.6 (which use the new hook available by Python and changes bytecode to add calls to the debugger so that there's less overhead during the debugging -- note that this only really takes place if breakpoints are added before a given code is loaded, adding or removing breakpoints afterwards falls back to the previous approach of tracing).

Another nice feature in this release is that isort (https://github.com/timothycrosley/isort) can be used as the default engine for sorting imports (needs to be configured in preferences > PyDev > Editor > Code Style > Imports -- note that at that same preferences dialog you may save the settings to a project, not only globally).

There were also a number of bug-fixes... in particular one that prevented text searches from working if the user had another plugin which also used Lucene in a different version was really nasty... http://www.pydev.org has more details on the changes.

This is also the first release which is signed with a proper certificate (provided by Comodo) -- so, it's nice that Eclipse won't complain that the plugin is not signed when it's being installed, although I discovered that it isn't as useful as I thought... it does work as intended for Eclipse plugins, but for Windows, even signing the LiClipse installer will show a dialog for users (there's a more expensive version with extended validation which could be used, but I didn't go for that one) and on Mac OS I haven't even tried to sign as it seems Comodo certificates are worthless there (the only choice is having a development subscription from Apple and using a certificate Apple gives you... the verification they do seems compatible with what Comodo gives, which uses a DUNS number, so, it's apparently just a point of them wanting more $$$/control, not really being more secure), so, currently Mac users will still use unsigned binaries (the sha256 is provided for users which want to actually check that what they download is what's being distributed).

mu-repo: Dealing with multiple git repositories

It's been a while since I've commented about mu-repo, so, now that 1.6.0 is available, I decided to give some more details on the latest additions ;)

-- if you're reading this and don't know what mu-repo is, it's a tool (done in Python) which helps when dealing with multiple git repositories (providing a way to call git commands on multiple repositories at once, along some other bells and whistles). http://fabioz.github.io/mu-repo has more details.

The last 2 major things that were introduced where:

1. A workflow for creating code-reviews in multiple repositories at once.

2. The possibility of executing non-git commands on multiple repositories at once.

For #1, the command mu open-url was created. Mostly, it'll compare the current branch against a different branch and open browser tabs making replacements in the url passed with the name of the repository (http://fabioz.github.io/mu-repo/open_url has more info and examples on how to use this for common git hosting platforms).

For #2, it's possible to execute a given command in the multiple tracked repositories by using the mu sh command. Mostly, call mu sh and pass the command you want to issue in the multiple tracked repositories.

e.g.: calling mu sh python setup.py develop will call python setup.py develop on each of the tracked repository directories.

That's it... enjoy!

PyDev 5.8.0: Code Coverage fixes, IronPython debugging

PyDev 5.8.0 is now available for download.

This release fixed some issues regarding the code coverage integration, and adds support to using the code coverage when running tests using pytest.

There were also fixes in the debugger for IronPython, which had stopped working with IronPython due to its lack of sys._current_frames (although an important note is that IronPython 2.7.6 and 2.7.7 don't work with PyDev because of a critical issue in IronPython, so, either keep to IronPython 2.7.5 or use the development version).

This is also the first release to add a way for clients to hook in the debugger, so, it's possible to customize the representation of custom classes (see https://github.com/fabioz/PyDev.Debugger/tree/master/pydevd_plugins/extensions for more details).

There were also fixes in the PyLint integration, updating docstrings, finding __init__ on code-completion when it's resolved to a superclass, etc... See: http://www.pydev.org for more details.

Enjoy ;)

PyDev 5.7.0: PyLint integration and Jython debugging

PyDev 5.7.0 is now out. Among the major changes in this release is a much improved PyLint integration and a fix to the debugger which prevented it from working with Jython.

The major change on the PyLint integration is that now instead of doing it as a builder within PyDev (which would act on any file changed), PyLint uses the same structure that the PyDev code analysis uses.

This means that by default it'll run only on open files (so, it will run less frequently), while still being able to ask for a full analysis on all files below a folder.

Also, using Ctrl+1 on a line with PyLint errors will provide an option to ignore the PyLint error (in the same way it could already ignore a PyDev code analysis error) and if the same error is reported by PyDev and PyLint in the same line, only the one from PyDev will be shown.

-- http://www.pydev.org/manual_adv_pylint.html has more details on the changes.

Besides this, there were improvements using the assign parameters to attributes (which will no longer add assignments already available) and when there's already a docstring available in a method, an option to update the docstring to add the missing parameters is now presented (both actions are accessible through Ctrl+1 on a function def line).

Other changes may be seen at: http://www.pydev.org, where full release notes are available.

LiClipse 3.6.0 is already out with the integrated changes.

Thank you very much to all the PyDev supporters and Patrons (http://www.patreon.com/fabioz), who help to keep PyDev moving forward!

PyDev 5.6.0 released: faster debugger, improved type inference for super and pytest fixtures

PyDev 5.6.0 is now already available for download (and is already bundled in LiClipse 3.5.0).

There are many improvements on this version!

The major one is that the PyDev.Debugger got some attention and should now be 60%-100% faster overall -- in all supported Python versions (and that's on top of the improvements done previously).

This improvement was a nice example of trading memory vs speed (the major change done was that the debugger now has 2 new caches, one for saving whether a frame should be skipped or not and another to save whether a given line in a traced frame should be skipped or not, which enables the debugger to make much less checks on those occasions).

Also, other fixes were done in the debugger. Namely:

• the variables are now properly displayed when the interactive console is connected to a debug session;
• it's possible to select the Qt version for which QThreads should be patched for the debugger to work with (in preferences > PyDev > Debug > Qt Threads);
• fixed an issue where a native Qt signal is not callable message was raised when connecting a signal to QThread.started.
• fixed issue displaying variable (Ctrl+Shift+D) when debugging.

Note: from this version onward, the debugger will now only support Python 2.6+ (I believe there should be very few Python 2.5 users -- Python 2.6 itself stopped being supported in 2013, so, I expect this change to affect almost no one -- if someone really needs to use an older version of Python, it's always possible to get an older version of the IDE/debugger too). Also, from now on, supported versions are actually properly tested on the ci (2.6, 2.7 and 3.5 in https://travis-ci.org/fabioz/PyDev.Debugger and 2.7, 3.5 in https://ci.appveyor.com/project/fabioz/pydev-debugger).

The code-completion (Ctrl+Space) and find definition (F3) also had improvements and can now deal with the Python super (so, it's possible to get completions and go to the definition of a method declared in a superclass when using the super construct) and pytest fixtures (so, if you have a pytest fixture, you should now be able to have completions/go to its definition even if you don't add a docstring to the parameter saying its expected type).

Also, this release improved the support in third-party packages, so, coverage, pycodestyle (previously pep8.py) and autopep8 now use the latest version available. Also, PyLint was improved to use the same thread pool used in code-analysis and an issue in the Django shell was fixed when django >= 1.10.

And to finish, the preferences for running unit-tests can now be saved to the project or user settings (i.e.: preferences > PyDev > PyUnit > Save to ...) and an issue was fixed when coloring the matrix multiplication operator (which was wrongly recognized as a decorator).

Thank you very much to all the PyDev supporters and Patrons (http://www.patreon.com/fabioz), who help to keep PyDev moving forward and to JetBrains, which sponsored many of the improvements done in the PyDev.Debugger.

PyDev 5.5 released

The main features introduced in PyDev 5.5 are:

• Ctrl+Shift+Alt+O allows jumping directly to the last hyperlink in the console (which means that when you have some exception on the console, it can be used to got directly to the error location without using a mouse).

• Ctrl+2, sw switches the target and value in an assignment (but may not work properly if more than one '=' char is found in the line).

• The code-completion which adds a local import can now be configured to add the local import to the top of the method, not only in the line above the current line (to use it, request a code-completion for some token which needs to be imported, then press tab to focus the completion pop-up and apply the completion with Shift pressed).

• Another improvement in code-completion is that it now properly supports method chaining and provides 'value' and 'name' fields when accessing enums.

Apart from those, multiple bug-fixes are also available (in refactoring, hovering on debug, parsing nested async calls on Py3 and grouping imports).

Enjoy!

p.s.: Thank you very much to all the PyDev supporters and Patrons (http://www.patreon.com/fabioz), who help to keep PyDev moving forward.

For LiClipse users, 3.4.0 is already available with the latest PyDev.