Using Pixi for environment management

[VeckoTheGecko]

What is your issue?

I've been scoping out using Pixi over at Parcels (OceanParcels/Parcels#2205). I was wondering whether this would also be something that xarray would be interested in as well. Our dev setup is quite similar to Xarray (i.e., conda based, with dynamic versioning).

Is this something that you have already considered/would be willing to have in the codebase? (couldn't find anything in the tracker) I think it can greatly simplify our CI/requirements, and also make it easy to run CI tests locally. It doesn't have to be promoted as the primary dev workflow.

I'm happy to contribute to this if its something we'd like here.

• Pixi docs
• Pydata Talk: Ruben Arts - A New Way to Manage Python Environments with Pixi

[dcherian]

Yes I think it'd be well worth the effort to switch to either pixi or uv.

[VeckoTheGecko]

Awesome! Pixi is a superset of uv (it uses uv on the backend for the pypi side of things AFAICT) but with conda support - so I think that would be the way forward.

[max-sixty]

I'm a big fan of uv! I use that for my own xarray work

(not familiar with pixi yet)

[VeckoTheGecko]

Since this would require some significant reworking of the CI/test running, what do you think would be sufficient acceptance criteria for a PR to ensure correctness?

I'm thinking no regression on test coverage. Is there anything else I should consider?

[dcherian]

yes we'd like all the current actions to work.

[keewis]

I do think this will be quite the effort, especially since pixi has not really been adapted for use with libraries yet (or at least that's what the devs told me when I talked to them at scipy this year). I've somewhat recently migrated pint-xarray to pixi (which has a CI setup that is pretty similar to xarray), and in particular the nightly CI has been a pain. Additionally, I have not yet figured out how to best deal with the lock file: it does make the environment creation insanely fast, but also brings its own set of issues.

(It also means that we need xarray-contrib/minimum-dependency-versions#30)

[VeckoTheGecko]

Additionally, I have not yet figured out how to best deal with the lock file: it does make the environment creation insanely fast, but also brings its own set of issues.

Wouldnt it be a matter of generating the lock file in CI and caching it (with the cache expiring each day)? https://github.com/actions/cache That can be its own build step (loading cache/generating if not already cached). I'm sure that would significantly speed things up, or is there something else I'm not considering?

(It also means that we need xarray-contrib/minimum-dependency-versions#30)

Thanks for linking this. At the moment I'm just keeping the environment files separately for min dep checking.

[dcherian]

I believe @maxrjones has a bunch of experience here, so maybe he can give us some pointers.

[max-sixty]

what's the case for Pixi vs uv?

we don't use either of two items here, IIUC? (or "multiple languages" covers conda-specific dependencies?)

[keewis]

we have binary dependencies (the backends, in particular), which are usually much easier to install from conda-forge. (I also think the UX of pixi is a bit nicer than that of uv, but that might just be because I spent much more time with pixi)

[VeckoTheGecko]

what's the case for Pixi vs uv?

• Composable environments (which I don't think uv has at all) - this is the main motivation of the issue and will result in a large de-duplication of requirement specification in environments
• Task running (users are able to do pixi run pre-commit running pre-commit in a pixi environment without having to install it separately, also can do pixi run docs etc. without having to memorise commands) - this will unify the dev experience making it easy for new contributors and addressing PY007 in Address repo-review suggestions #8239

We do also have dependencies like opendap server which require gunicorn.

[max-sixty]

for me, the xarray-specific reasons — like dependencies which require conda — are more compelling than "pixi is generally better than uv", given the relative popularity of the two — new contributors are much more likely to have used uv than Pixi.

(but I'm not familiar with Pixi and if lots of us are really saying "honestly it's much much better than uv!!", then great...)