Feedback on the new resolution mechanism

[krassowski]

0. It looks like this is going to break a few more packages. I just tried running CI on jupyter-server and see this:

+ '[' Linux == Windows ']'
+ '[' -f pyproject.toml ']'
+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
error: Failed to download and build: overrides==0.1
  Caused by: Failed to build: overrides==0.1
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
--- stdout:

Also many other jobs (e.g. https://github.com/krassowski/jupyter_server/actions/runs/8542681567/job/23404855225) seem to be failing with:

E       TypeError: websocket_connect() got an unexpected keyword argument 'body'

but this might be unrelated

2. It installs the lowest version possible when no floor version is defined:

dependencies = [
    "async_lru>=1.0.0",
    "httpx>=0.25.0",
    "importlib-metadata>=4.8.3;python_version<\"3.10\"",
    "importlib-resources>=1.4;python_version<\"3.9\"",
    "ipykernel",
    "jinja2>=3.0.3",
    "jupyter_core",
    "jupyter_server>=2.4.0,<3",
    "jupyter-lsp>=2.0.0",
    "jupyterlab_server>=2.19.0,<3",
    "notebook_shim>=0.2",
    "packaging",
    "tomli;python_version<\"3.11\"",
    "tornado>=6.2.0",
    "traitlets",
]

Results in:

+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
error: Failed to download and build: traitlets==0.0.1
  Caused by: Failed to build: traitlets==0.0.1
  Caused by: Build backend failed to determine extra requires with `build_wheel()` with exit status: 1

3. It pins to oldest available version even if other dependencies of dependencies have a pin (after pinning traitlets to something modern):

+ uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
Resolved 92 packages in 1.07s
# This file was autogenerated by uv via the following command:
#    uv pip compile --resolution=lowest-direct pyproject.toml -o /home/runner/constraints.txt
anyio==4.3.0
    # via
    #   httpcore
    #   jupyter-server
argon2-cffi==23.1.0
    # via jupyter-server
argon2-cffi-bindings==21.2.0
    # via argon2-cffi
arrow==1.3.0
    # via isoduration
asttokens==2.4.1
    # via stack-data
async-lru==1.0.0
attrs==23.2.0
    # via
    #   jsonschema
    #   referencing
babel==2.14.0
    # via jupyterlab-server
backcall==0.2.0
    # via ipython
beautifulsoup4==4.12.3
    # via nbconvert
bleach==6.1.0
    # via nbconvert
certifi==2024.2.2
    # via
    #   httpcore
    #   httpx
    #   requests
cffi==1.16.0
    # via argon2-cffi-bindings
charset-normalizer==3.3.2
    # via requests
decorator==5.1.1
    # via ipython
defusedxml==0.7.1
    # via nbconvert
exceptiongroup==1.2.0
    # via anyio
executing==2.0.1
    # via stack-data
fastjsonschema==2.19.1
    # via nbformat
fqdn==1.5.1
    # via jsonschema
h11==0.14.0
    # via httpcore
httpcore==0.18.0
    # via httpx
httpx==0.25.0
idna==3.6
    # via
    #   anyio
    #   httpx
    #   jsonschema
    #   requests
importlib-metadata==4.8.3
    # via
    #   jupyter-client
    #   jupyter-lsp
    #   jupyterlab-server
    #   nbconvert
importlib-resources==1.4.0
    # via
    #   jsonschema
    #   jsonschema-specifications
ipykernel==4.0.1
ipython==8.12.3
    # via ipykernel
isoduration==20.11.0
    # via jsonschema
jedi==0.19.1
    # via ipython
jinja2==3.0.3
    # via
    #   jupyter-server
    #   jupyterlab-server
    #   nbconvert
json5==0.9.24
    # via jupyterlab-server
jsonpointer==2.4
    # via jsonschema
jsonschema==4.21.1
    # via
    #   jupyter-events
    #   jupyterlab-server
    #   nbformat
jsonschema-specifications==2023.12.1
    # via jsonschema
jupyter-client==8.6.1
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
jupyter-core==4.12.0
    # via
    #   jupyter-client
    #   jupyter-server
    #   nbclient
    #   nbconvert
    #   nbformat
jupyter-events==0.10.0
    # via jupyter-server
jupyter-lsp==2.0.0
jupyter-server==2.4.0
    # via
    #   jupyter-lsp
    #   jupyterlab-server
    #   notebook-shim
jupyter-server-terminals==0.5.3
    # via jupyter-server
jupyterlab-pygments==0.3.0
    # via nbconvert
jupyterlab-server==2.19.0
markupsafe==2.1.5
    # via
    #   jinja2
    #   nbconvert
matplotlib-inline==0.1.6
    # via ipython
mistune==3.0.2
    # via nbconvert
nbclient==0.10.0
    # via nbconvert
nbconvert==7.16.3
    # via jupyter-server
nbformat==5.10.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
notebook-shim==0.2.0
packaging==21.3
    # via
    #   jupyter-server
    #   jupyterlab-server
    #   nbconvert
pandocfilters==1.5.1
    # via nbconvert
parso==0.8.3
    # via jedi
pexpect==4.9.0
    # via ipython
pickleshare==0.7.5
    # via ipython
pkgutil-resolve-name==1.3.10
    # via jsonschema
prometheus-client==0.20.0
    # via jupyter-server
prompt-toolkit==3.0.43
    # via ipython
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pure-eval==0.2.2
    # via stack-data
pycparser==2.22
    # via cffi
pygments==2.17.2
    # via
    #   ipython
    #   nbconvert
pyparsing==3.1.2
    # via packaging
python-dateutil==2.9.0.post0
    # via
    #   arrow
    #   jupyter-client
python-json-logger==2.0.7
    # via jupyter-events
pytz==2024.1
    # via babel
pyyaml==6.0.1
    # via jupyter-events
pyzmq==25.1.2
    # via
    #   jupyter-client
    #   jupyter-server
referencing==0.34.0
    # via
    #   jsonschema
    #   jsonschema-specifications
    #   jupyter-events
requests==2.31.0
    # via jupyterlab-server
rfc3339-validator==0.1.4
    # via
    #   jsonschema
    #   jupyter-events
rfc3986-validator==0.1.1
    # via
    #   jsonschema
    #   jupyter-events
rpds-py==0.18.0
    # via
    #   jsonschema
    #   referencing
send2trash==1.8.2
    # via jupyter-server
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   python-dateutil
    #   rfc3339-validator
sniffio==1.3.1
    # via
    #   anyio
    #   httpcore
    #   httpx
soupsieve==2.5
    # via beautifulsoup4
stack-data==0.6.3
    # via ipython
terminado==0.18.1
    # via
    #   jupyter-server
    #   jupyter-server-terminals
tinycss2==1.2.1
    # via nbconvert
tomli==0.2.0
tornado==6.2
    # via
    #   jupyter-client
    #   jupyter-server
    #   terminado
traitlets==5.6.0
    # via
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-events
    #   jupyter-server
    #   matplotlib-inline
    #   nbclient
    #   nbconvert
    #   nbformat
types-python-dateutil==2.9.0.20240316
    # via arrow
typing-extensions==4.10.0
    # via
    #   anyio
    #   ipython
uri-template==1.3.0
    # via jsonschema
urllib3==2.2.1
    # via requests
wcwidth==0.2.13
    # via prompt-toolkit
webcolors==1.13
    # via jsonschema
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.7.0
    # via jupyter-server
zipp==3.18.1
    # via importlib-metadata

/scripts/ci_install.sh: line 19: yarn: command not found
+ mkdir -p /home/runner/.jupyter
+ git config --global user.name foo
+ git config --global user.email foo@bar.com
+ pip install -q --upgrade pip --user
+ pip --version
pip 24.0 from /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip (python 3.8)
+ pip install -e '.[dev,test]'
Obtaining file:///home/runner/work/jupyterlab/jupyterlab
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [[29](https://github.com/krassowski/jupyterlab/actions/runs/8542572841/job/23404523285#step:4:30) lines of output]
      Collecting hatchling>=1.21.1
        Using cached hatchling-1.22.4-py3-none-any.whl.metadata (3.8 kB)
      Collecting packaging>=21.3 (from hatchling>=1.21.1)
        Downloading packaging-21.3-py3-none-any.whl.metadata (15 kB)
      Collecting pathspec>=0.10.1 (from hatchling>=1.21.1)
        Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
      Collecting pluggy>=1.0.0 (from hatchling>=1.21.1)
        Using cached pluggy-1.4.0-py3-none-any.whl.metadata (4.3 kB)
      INFO: pip is looking at multiple versions of hatchling to determine which version is compatible with other requirements. This could take a while.
      Collecting hatchling>=1.21.1
        Downloading hatchling-1.22.3-py3-none-any.whl.metadata (3.8 kB)
        Downloading hatchling-1.22.2-py3-none-any.whl.metadata (3.8 kB)
        Using cached hatchling-1.21.1-py3-none-any.whl.metadata (3.8 kB)
      Collecting editables>=0.3 (from hatchling>=1.21.1)
        Using cached editables-0.5-py3-none-any.whl.metadata (3.1 kB)
      ERROR: Cannot install hatchling==1.21.1, hatchling==1.22.2, hatchling==1.22.3 and hatchling==1.22.4 because these package versions have conflicting dependencies.
      
      The conflict is caused by:
          hatchling 1.22.4 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.22.3 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.22.2 depends on tomli>=1.2.2; python_version < "3.11"
          hatchling 1.21.1 depends on tomli>=1.2.2; python_version < "3.11"
          The user requested (constraint) tomli==0.2.0
      
      To fix this you could try to:
      1. loosen the range of package versions you've specified
      2. remove package versions to allow pip attempt to solve the dependency conflict
      
      ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
+ pip install -v -e '.[dev,test]'
Using pip 24.0 from /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pip (python 3.8)
Obtaining file:///home/runner/work/jupyterlab/jupyterlab
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting hatchling>=1.21.1
    Using cached hatchling-1.22.4-py3-none-any.whl.metadata (3.8 kB)
  Collecting packaging>=21.3 (from hatchling>=1.21.1)
    Using cached packaging-21.3-py3-none-any.whl.metadata (15 kB)
  Collecting pathspec>=0.10.1 (from hatchling>=1.21.1)
    Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
  Collecting pluggy>=1.0.0 (from hatchling>=1.21.1)
    Using cached pluggy-1.4.0-py3-none-any.whl.metadata (4.3 kB)
  INFO: pip is looking at multiple versions of hatchling to determine which version is compatible with other requirements. This could take a while.
  Collecting hatchling>=1.21.1
    Using cached hatchling-1.22.3-py3-none-any.whl.metadata (3.8 kB)
    Using cached hatchling-1.22.2-py3-none-any.whl.metadata (3.8 kB)
    Using cached hatchling-1.21.1-py3-none-any.whl.metadata (3.8 kB)
  Collecting editables>=0.3 (from hatchling>=1.21.1)
    Using cached editables-0.5-py3-none-any.whl.metadata (3.1 kB)
  ERROR: Cannot install hatchling==1.21.1, hatchling==1.22.2, hatchling==1.22.3 and hatchling==1.22.4 because these package versions have conflicting dependencies.
  The conflict is caused by:
      hatchling 1.22.4 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.22.3 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.22.2 depends on tomli>=1.2.2; python_version < "3.11"
      hatchling 1.21.1 depends on tomli>=1.2.2; python_version < "3.11"
      The user requested (constraint) tomli==0.2.0
  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict
  ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
  error: subprocess-exited-with-error

[krassowski]

Also on notebook in jupyter/notebook#7312:

+ '[' standard == pre ']'
+ '[' standard == minimum ']'
+ uv pip compile pyproject.toml -o /home/runner/constraints.txt
Resolved 91 packages in 559ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml -o /home/runner/constraints.txt
anyio==4.3.0
    # via
    #   httpx
    #   jupyter-server
argon2-cffi==23.1.0
    # via jupyter-server
argon2-cffi-bindings==21.2.0
    # via argon2-cffi
arrow==1.3.0
    # via isoduration
asttokens==2.4.1
    # via stack-data
async-lru==2.0.4
    # via jupyterlab
attrs==23.2.0
    # via
    #   jsonschema
    #   referencing
babel==2.14.0
    # via jupyterlab-server
beautifulsoup4==4.12.3
    # via nbconvert
bleach==6.1.0
    # via nbconvert
certifi==2024.2.2
    # via
    #   httpcore
    #   httpx
    #   requests
cffi==1.16.0
    # via argon2-cffi-bindings
charset-normalizer==3.3.2
    # via requests
comm==0.2.2
    # via ipykernel
debugpy==1.8.1
    # via ipykernel
decorator==5.1.1
    # via ipython
defusedxml==0.7.1
    # via nbconvert
exceptiongroup==1.2.0
    # via
    #   anyio
    #   ipython
executing==2.0.1
    # via stack-data
fastjsonschema==2.19.1
    # via nbformat
fqdn==1.5.1
    # via jsonschema
h11==0.14.0
    # via httpcore
httpcore==1.0.5
    # via httpx
httpx==0.27.0
    # via jupyterlab
idna==3.6
    # via
    #   anyio
    #   httpx
    #   jsonschema
    #   requests
ipykernel==6.29.4
    # via jupyterlab
ipython==8.23.0
    # via ipykernel
isoduration==20.11.0
    # via jsonschema
jedi==0.19.1
    # via ipython
jinja2==3.1.3
    # via
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
json5==0.9.24
    # via jupyterlab-server
jsonpointer==2.4
    # via jsonschema
jsonschema==4.21.1
    # via
    #   jupyter-events
    #   jupyterlab-server
    #   nbformat
jsonschema-specifications==2023.12.1
    # via jsonschema
jupyter-client==8.6.1
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
jupyter-core==5.7.2
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   nbclient
    #   nbconvert
    #   nbformat
jupyter-events==0.10.0
    # via jupyter-server
jupyter-lsp==2.2.4
    # via jupyterlab
jupyter-server==2.13.0
    # via
    #   jupyter-lsp
    #   jupyterlab
    #   jupyterlab-server
    #   notebook-shim
jupyter-server-terminals==0.5.3
    # via jupyter-server
jupyterlab==4.2.0b0
jupyterlab-pygments==0.3.0
    # via nbconvert
jupyterlab-server==2.25.4
    # via jupyterlab
markupsafe==2.1.5
    # via
    #   jinja2
    #   nbconvert
matplotlib-inline==0.1.6
    # via
    #   ipykernel
    #   ipython
mistune==3.0.2
    # via nbconvert
nbclient==0.10.0
    # via nbconvert
nbconvert==7.16.3
    # via jupyter-server
nbformat==5.10.3
    # via
    #   jupyter-server
    #   nbclient
    #   nbconvert
nest-asyncio==1.6.0
    # via ipykernel
notebook-shim==0.2.4
    # via jupyterlab
overrides==7.7.0
    # via jupyter-server
packaging==24.0
    # via
    #   ipykernel
    #   jupyter-server
    #   jupyterlab
    #   jupyterlab-server
    #   nbconvert
pandocfilters==1.5.1
    # via nbconvert
parso==0.8.3
    # via jedi
pexpect==4.9.0
    # via ipython
platformdirs==4.2.0
    # via jupyter-core
prometheus-client==0.20.0
    # via jupyter-server
prompt-toolkit==3.0.43
    # via ipython
psutil==5.9.8
    # via ipykernel
ptyprocess==0.7.0
    # via
    #   pexpect
    #   terminado
pure-eval==0.2.2
    # via stack-data
pycparser==2.22
    # via cffi
pygments==2.17.2
    # via
    #   ipython
    #   nbconvert
python-dateutil==2.9.0.post0
    # via
    #   arrow
    #   jupyter-client
python-json-logger==2.0.7
    # via jupyter-events
pyyaml==6.0.1
    # via jupyter-events
pyzmq==25.1.2
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
referencing==0.34.0
    # via
    #   jsonschema
    #   jsonschema-specifications
    #   jupyter-events
requests==2.31.0
    # via jupyterlab-server
rfc3339-validator==0.1.4
    # via
    #   jsonschema
    #   jupyter-events
rfc3986-validator==0.1.1
    # via
    #   jsonschema
    #   jupyter-events
rpds-py==0.18.0
    # via
    #   jsonschema
    #   referencing
send2trash==1.8.2
    # via jupyter-server
six==1.16.0
    # via
    #   asttokens
    #   bleach
    #   python-dateutil
    #   rfc3339-validator
sniffio==1.3.1
    # via
    #   anyio
    #   httpx
soupsieve==2.5
    # via beautifulsoup4
stack-data==0.6.3
    # via ipython
terminado==0.18.1
    # via
    #   jupyter-server
    #   jupyter-server-terminals
tinycss2==1.2.1
    # via nbconvert
tomli==2.0.1
    # via jupyterlab
tornado==6.4
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   terminado
traitlets==5.14.2
    # via
    #   comm
    #   ipykernel
    #   ipython
    #   jupyter-client
    #   jupyter-core
    #   jupyter-events
    #   jupyter-server
    #   jupyterlab
    #   matplotlib-inline
+ echo PIP_CONSTRAINT=/home/runner/constraints.txt
    #   nbclient
    #   nbconvert
    #   nbformat
types-python-dateutil==2.9.0.20240[316](https://github.com/jupyter/notebook/actions/runs/8539292958/job/23396548527?pr=7312#step:3:330)
    # via arrow
typing-extensions==4.10.0
    # via
    #   anyio
    #   async-lru
    #   ipython
uri-template==1.3.0
    # via jsonschema
urllib3==2.2.1
    # via requests
wcwidth==0.2.13
    # via prompt-toolkit
webcolors==1.13
    # via jsonschema
webencodings==0.5.1
    # via
    #   bleach
    #   tinycss2
websocket-client==1.7.0
    # via jupyter-server

+ hatch run cov:test
Creating environment: cov
Installing project in development mode
ERROR: Cannot install jupyter-server, jupyterlab-server and openapi-core because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

https://github.com/jupyter/notebook/actions/runs/8539292958/job/23396548527?pr=7312

[krassowski]

I guess there is nothing to do here. With bumping a few pins the job is now passing for me. I just think that maybe this should have been a new release given how many downstreams are affected? But maybe it is ok if it does find at least one true positive where lack of a pin made a difference (so far I have not seen it though).

[jtpio]

Yes, and it's also a bit tedious to fix. For example in the case of the notebook issue reported above, the default hatch run cov:test does not give enough information:

+ hatch run cov:test
Creating environment: cov
Installing project in development mode
ERROR: Cannot install jupyter-server, jupyterlab-server and openapi-core because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

While doing a plain pip install -ve ".[dev,test]" gives more pointers:

In this case, it looks like it's not even fixable in notebook, but requires a fix in jupyterlab_server first?

[jtpio]

In this case, it looks like it's not even fixable in notebook, but requires a fix in jupyterlab_server first?

OK the jupyterlab_server CI is also failing: jupyterlab/jupyterlab_server#446

So it looks like that would mean fixing the dependencies in jupyterlab_server, and then making a new release of jupyterlab_server to fix the CI of other downstreams (after bumping to the latest jupyterlab_server)?

[krassowski]

Yes, the thing worries me is that even if we do some magic with the pins the final resolution may not represent the actual resolution users get when installing using pip. I think using the new constraints file generation method for minimum resolution mode is fine, but by default we should not use it.

[blink1073]

1. That was a separate bug that I fixed.
2. I think that is actually a good thing, and I've been updating packages to be more accurate.

I think the errors we're seeing in practice will be fixed by using --extra test in the resolution, doing that now.

[blink1073]

I think using the new constraints file generation method for minimum resolution mode is fine, but by default we should not use it.

I disagree, this is an "advanced" check, and we're making it a bit stricter and more correct.

[blink1073]

Ah, I do see the point about affecting regular installs. If the next release doesn't fix that in most cases, we can make it opt-in for regular installs.

[blink1073]

Okay, no, now jupyter_server doesn't install. Let's call this a failed experiment...