PR: Auto-install spyder-kernels into external environment

[mrclary]

Description of Changes

When the selected Python interpreter environment does not have a compatible spyder-kernels version installed, offer to install it.
If the user rejects the installation, then the conventional error is displayed.
If the user accepts the installation, then spyder-kernels is installed into the environment from the same channel as the environment Python.
Upon successful completion, the kernel is automatically connected.

Issue(s) Resolved

Fixes #22855

[dalthviz]

Checked this locally on Windows and for the conda case seems like things are working 👍

Also some notes from my check:

• Noticed that you could potencially accept installing spyder-kernels from multiple consoles and sometimes you can briefly see a loading widget alongside the actual console widget (not sure if having multiple consoles installing could be the cause of that behavior 🤔):

• Seems like if you have multiple consoles before spyder-kernels is installed (even when an installation was accepted) you can potencially close all the consoles or create new ones which you will need to end up closing manually to be able to detect that spyder-kernels was installed:

[mrclary]

Checked this locally on Windows and for the conda case seems like things are working 👍

Thanks for checking on this, @dalthviz.

• Noticed that you could potencially accept installing spyder-kernels from multiple consoles

I assume that you mean multiple consoles connected to the same kernel or using the same environment. I'm not sure what to do about that. Since this is being handled by the client, multiple clients using the same environment could independently offer to install spyder-kernels. Maybe some kind of check could be run through the main widget that would prevent multiple offerings on the same environment?

and sometimes you can briefly see a loading widget alongside the actual console widget (not sure if having multiple consoles installing could be the cause of that behavior 🤔):

I don't think this has anything to do with multiple consoles. I'm glad that you saw this. I think I saw this as well, but not sure how it's happening, since each individual widget (shellwidget, infowidget, and installwidget) is supposed to be hidden before a particular widget is made visible. Hmm...

• Seems like if you have multiple consoles before spyder-kernels is installed (even when an installation was accepted) you can potencially close all the consoles or create new ones which you will need to end up closing manually to be able to detect that spyder-kernels was installed:

I see. Let's talk about the desired UX with the group and see what we can do about it.

[CAM-Gerlach]

UI wise, the message looks rather different from the other messages we show in the console, in terms of layout, background, text, etc, where it looks more like regular console content than the other pane messages we show (errors, empty pane messages, help pane empty message, etc). Maybe for a followup PR for Carlos/Andres to work on, but perhaps some low hanging fruit fixes would be practical in this PR, e.g. centering the text and perhaps adding a border like the other error/informational messages.

Also, to be clearer that clicking the "OK" button will actually execute the install on their environment rather than merely just accepting that the environment is outdated, you might want to rename it "Install". Or, perhaps to further clarify what the other option is and that it isn't just giving up, the options could be "Install automatically" and "Install manually"?

I assume that you mean multiple consoles connected to the same kernel or using the same environment. I'm not sure what to do about that. Since this is being handled by the client, multiple clients using the same environment could independently offer to install spyder-kernels. Maybe some kind of check could be run through the main widget that would prevent multiple offerings on the same environment?

It does seem to me that this scenario isn't that likely in practice for actual users, since they are going to get the message on the first console they start in that environment and presumably going to decide to either accept and let it finish or dismiss it before opening another one in the same environment, expecting different results. However, it does seem worth at least making sure that unexpected behavior doesn't result if the user requests multiple installations, e.g. by disabling the "OK" button or not offering the installation if the user opens another console, though it would be better to show a dedicated message that an install is in progress and ideally auto-restart the console when it completes.

[mrclary]

I don't know why the tests are taking so long and result in being cancelled. The failing tests pass locally on my Windows machine 🤷🏼 .

[mrclary]

TL;DR

I have found a potential problem with auto-installing spyder-kernels in external environments using conda. The environment solve may fail or it may produce incorrect results under common circumstances. However, mamba appears to work flawlessly.

conda can be used to install spyder-kernels into external environments if neither spyder-kernels nor its dependencies are already installed. This could be a serious limitation. mamba should work reliably for any circumstance, provided it is available. This should not be a problem for our installers, in which we can ensure that mamba is available. However, for standard conda installs of Spyder, mamba may not be available. We may need to instruct the user to manually install spyder-kernels in this circumstance.

The Problem

To demonstrate the issue I created three separate environments:

• A: python=3.12, numpy=2.1
• B: python=3.12, numpy=2.1, ipython=8.13
• C: python=3.12, numpy=2.1, ipython=8.13, spyder-kernels=3.0

These represent three general scenarios: when neither spyder-kernels nor any dependencies are installed; when spyder-kernels is not installed but a dependency is installed; and when spyder-kernels and its dependencies are already installed. numpy represents an unrelated package and ipython represents a spyder-kernel dependency.

These environments are updated using the following command.

conda install -n <env> "spyder-kernels>=3.1,<3.2"

We require updating the environment by installing the correct version of spyder-kernels, with compatible dependencies, but without changing unrelated packages.

Updating environment A succeeds without any issue. However, environments B and C fail to solve:

Could not solve for environment specs
The following packages are incompatible
├─ ipython =8.13 * is installable with the potential options
│  ├─ ipython [8.13.1|8.13.2|...|8.18.0] would require
│  │  └─ __linux =* *, which is missing on the system;
│  ├─ ipython [8.13.1|8.13.2|...|9.9.0] would require
│  │  └─ __win =* *, which is missing on the system;
│  └─ ipython [8.13.1|8.13.2], which can be installed;
└─ spyder-kernels >=3.1,<3.2 * is not installable because there are no viable options
   ├─ spyder-kernels [3.1.0|3.1.1|3.1.2] would require
   │  └─ ipython >=8.15.0,(<10.0.0,(!=8.17.1,(!=9.1.0,(!=9.2.0,(!=9.3.0,!=9.4.0))))) * but there are no viable options
   │     ├─ ipython [8.13.1|8.13.2|...|8.18.0], which cannot be installed (as previously explained);
   │     ├─ ipython [8.15.0|8.16.0|...|9.9.0] conflicts with any installable versions previously reported;
   │     └─ ipython [8.13.1|8.13.2|...|9.9.0], which cannot be installed (as previously explained);
   └─ spyder-kernels [3.1.0|3.1.1|3.1.2] would require
      └─ __win =* *, which is missing on the system.

The solve apparently cannot accommodate the required update for ipython.

The update can be successful if the --update-all flag is used, but this has the undesirable affect of updating numpy as well. Using other solver mode modifier flags does not change the outcome: --only-deps, --no-pin, --update-deps, --update-specs.

Using conda update rather than conda install produces effectively the same results, but has some notable differences. For environments A and B, an exception is raised because spyder-kernels is not installed, so this is effectively the same as a failed solve. For environment C, rather than a failed solve, no update is performed and conda reports that all the required packages are already installed. Note that conda update does not permit specifying version constraints, so the outcome of updating environment C is expected. However, using the --update-deps flag results in a successful solve that correctly updates ipython but downgrades spyder-kernels to version 0.2.6! Interestingly, the downgraded version changes depending on the Python version in the environment, but I could not find a scenario in which this resulted in an actual upgrade to spyder-kernels. Nevertheless, the inability to accommodate a version specification additionally precludes using the update command since this, even if successful, would not ensure the correct version of spyder-kernels for the user's installed version of Spyder (e.g. the user has an older version of Spyder).

I did find, however, that mamba install performs perfectly in all three environment scenarios.

All the above was investigated on both macOS and Windows with identical results on both machines.

Is this issue specific to ipython? Will conda install succeed in (most) other circumstances? Will mamba install fail under any circumstances? Unfortunately, I don't know the answer to these questions and believe the only way to find out is to implement the feature and wait for user feedback. I do think it likely that ipython is a good test dependency because it has a high probability of being installed for other reasons. Additionally, spyder-kernels's dependency specification for it is rather complex: ipython >=8.15.0,<10.0.0,!=8.17.1,!=9.1.0,!=9.2.0,!=9.3.0,!=9.4.0. So I think it reasonable to believe that if a solution works for ipython, it will work generally.