Revert to flit backend for improved cross-platform experience

[ds-cbo]

As briefly discussed in #414, I suggested moving back to pypa/flit to significantly improve user experience on environments where uv-build is not suited. For example because compilation times exceeds the 15 minutes or it does not work[1], or where its dependency tree of 650+ crates does not pass your risk assessment for potential supply chain attacks compared to flit-core's single vendored package. The latter being especially relevant since cachecontrol is a dependency of pypa/pip-audit, common in security-oriented environments.

Note that the uv build command as used in make dist does not need to be changed, nor does build have to be installed, as:

uv supports all build backends (as specified by PEP 517)
(source)

This is a partial revert of 02df424 labeled "chore(ci): cleanup, add Python 3.14" which changed the build backend from flit-core to uv-build. Note that flit also has support for Python 3.14 (and 3.15).

I compared the resulting tarballs and wheels and they were practically equal.

[1] uv-build only supports few architectures in Tier 1 "guaranteed to work", less than Python itself - https://docs.astral.sh/uv/reference/policies/platforms/ vs https://peps.python.org/pep-0011/

[woodruffw]

I'm pretty hesitant to revert this -- the median performance benefit is significant (most users aren't building the uv_build backend from scratch), and the dependency stack in question is already present (it's the same stack that the uv CLI is built against).

My medium-term preference is to solve this through compounding, since I think distros will need to handle uv's build backend eventually anyways. In other words, if the problem is that CacheControl uses uv-build but pip-audit uses flit, I'm inclined to change pip-audit to use uv-build too.

I'm curious if the above would cause problems for you, or whether it's just the first build of uv-build that's problematic.

[ds-cbo]

the dependency stack in question is already present

In our distro's build environment we use the general pypa/build frontend to build wheels, so we avoid having to build and install the uv frontend, which is more of a development tool as far as I understand it. So no, I do not agree that it is already present anyway.

This environment is also security-critical, so we do extensive checks on the tools that enter it and disallow any binary blobs. This is why we would prefer to avoid having the 650+ dependencies of uv-build, especially if we know that alternatives like flit-core with 0 dependencies also gets the task done in the correct way.

I think distros will need to handle uv's build backend eventually anyways

It's not that distros don't support uv (it builds fine after 15 minutes, and then builds cachecontrol fine as well). If anything, it seems to be quite the contrary with uv not supporting several platforms (notably any *BSD), meaning it's just "an accident" that it does seem to work, and there are no guarantees for future upgrades.

if the problem is that CacheControl uses uv-build but pip-audit uses flit, I'm inclined to change pip-audit to use uv-build too.

The fragmentation of new python build tooling (hatchling, flit-core, pdm-backend, most recently uv-build, perhaps more I don't know of) is indeed part of the reason why it's getting harder to package wheels for distros, but I think that uv-build in particular is making it harder considering my points raised before. For pure python packages such as cachecontrol, flit-core seems to me to be the smallest and most simple tool to get the job done. At least in my experience that's what most similar packages use and I don't see any particular reason why it would not work in this case.

If you would prefer to keep cachecontrol on uv-build, I would rather prefer to change pip-audit to stop depending on cachecontrol to adapt to the security-conscious market which would prefer to keep the attack surface introduced by subdependencies small. That would seem to me like a choice fit for a tool such as pip-audit.

I'm curious if the above would cause problems for you, or whether it's just the first build of uv-build that's problematic.

My problems are two-fold both arriving from the fact that uv-build has 650+ dependencies: the first being that compilations need to happen often due to dependency upgrades and take long as a result of that; the second being that it introduces a large attack surface for supply chain attacks and is therefore not preferred in security-conscious environments.

It's not just "the first build" that is problematic, it's that every build is problematic, and those need to happen regularly.

[woodruffw]

So no, I do not agree that it is already present anyway.

Sorry, I meant: uv is present during the maintenance process, and you're implicitly trusting it because I'm using it to maintain CacheControl, pip-audit, etc. already. So it's already (for better or worse) in the "circle of trust."

If anything, it seems to be quite the contrary with uv not supporting several platforms (notably any *BSD), meaning it's just "an accident" that it does seem to work, and there are no guarantees for future upgrades.

To my understanding, this is due primarily a lack of reliable CI and build environments that uv can perform BSD builds on, plus (?) limited wheel tag support for BSDs. In general uv is interested in adding support for any platform that can be reliably tested against.

That would seem to me like a choice fit for a tool such as pip-audit.

The main reason pip-audit uses CacheControl is because it reuses pip's cache entries as an optimization. We could discuss on the pip-audit tracker but I'd be similarly hesitant to remove that optimization, since auditing N dependencies otherwise requires N network cycles and users find the resulting slowness frustrating.

(In case I left it implicit: I maintain both pip-audit and CacheControl, so I'm not super worried about supply chain sprawl between the two -- they have the same underlying maintainer.)

My problems are two-fold both arriving from the fact that uv-build has 650+ dependencies: the first being that compilations need to happen often due to dependency upgrades and take long as a result of that; the second being that it introduces a large attack surface for supply chain attacks and is therefore not preferred in security-conscious environments.

Sorry, I think I'm not understanding why rebuilds happen so often: uv itself provides a lockfile, and there's no reason to preemptively invalidate your builds if the existing dependencies are range-compatible. Is this because you're building globally unique versions of each crate, and thus need to rebuild whenever one of those bumps?

I think this entire situation is unfortunate and I sympathize with the problems it causes for your distro setup, but I think the long-term reality is that more packages will use uv-build and Python's build backend proliferation won't really stop (this is not ideal IMO, but is essentially a coordination and standards-over-implementation problem that's inherent to Python packaging).

Given that, I think there are two options here:

1. You can keep the uv-build backend, which I understand is not ideal for the aforementioned reasons. This will hopefully be amortized over time since other packages are continuing to adopt it.
2. You can carry a local patch (the one in this PR) for your distro-specific builds. I suspect that patch will work for a long while and I won't try to break it, but it also won't be officially supported upstream. My understanding (which I apologize if wrong) is that it's somewhat common for distros to carry these kinds of patches, since Python upstreams frequently have Python packaging-isms in them that don't apply to distro builds.

[woodruffw]

xref astral-sh/uv#3370 for the uv FreeBSD build tracking.

[ds-cbo]

First of all thanks for your extensive answer, I'm glad to see you take my input seriously :)

So it's already (for better or worse) in the "circle of trust."

Sortof, if it were to misbehave on your machine we would not notice it directly, unless it installs a worm of some kind that spreads through its generated sdist files but does not show up in the git tag diffs. Supply chain attacks can take many forms, but as far as I see them it usually just steals data from the machine it runs on, and in this case that would only be the one of the maintainer who builds the sdist, not the one who later builds the sdist - unless it also requires the same tool as build dependency of course.

this is due primarily a lack of reliable CI and build environments that uv can perform BSD builds on

I will sadly have to agree with you on this one, the BSDs are indeed not too popular in CI environments, so I understand the caution of anything that requires a compiler. Which is also why we prefer running Pure Python on our systems, even though the Rust compiler itself does have FreeBSD support. I don't want to blame uv itself for this, as it's usually only for developers to use instead of build servers, but uv-build is the bridge that makes it inconvenient here.

The main reason pip-audit uses CacheControl is because it reuses pip's cache entries as an optimization

My suggestion to remove it was indeed a bit extreme, I would've imagined a replacement fork.

(In case I left it implicit: I maintain both pip-audit and CacheControl

I was aware, that's why I mentioned it as an example. I'm also aware you work for the company behind uv, which is why I wasn't too hopeful of this PR, but I wanted to raise my concerns anyway.

uv itself provides a lockfile [...] Is this because you're building globally unique versions of each crate, and thus need to rebuild whenever one of those bumps?

I had to double check this, because I thought that was indeed the case, but it does seem to be using the lockfile and independent crate installs (source). That would be explained by the compile times, since it cannot reuse anything from elsewhere. That means it should only trigger a rebuild when a dynamically linked library (libgit2, libssh2, libzstd), the rust compiler itself or other build tools (gmake, perl) upgrade, which might've also been the trigger last few times I did system upgrades.

You can carry a local patch (the one in this PR) for your distro-specific builds

That was indeed the approach I imagined would be the final result, which happens to be easier if you already have a working PR.

I won't try to break it

That's most appreciated, having to deal with these "Python packaging-isms" is indeed one of my main annoyances maintaining a bunch of distro packages in a security-conscious environment nowadays. setuptools wasn't perfect, but the modern attempts-of-solutions aren't making it any easier 😞

[woodruffw]

Thank you for your answers too! I'm sorry my answer isn't as simple as a yes, but I appreciate that the entire situation around Python packaging is frustrating from a distro perspective. I also appreciate the context you've brought around FreeBSD and gaps in uv's support; that's admittedly a blind spot of mine (and not something that Python packaging has a ton of insight into, since most user reports bubble up from user-level rather than distro-level packaging).

I'd definitely love to push on uv's FreeBSD support -- it's possible that real progress could be made on it given some of the non-GHA CIs out there, although I think stability/testability will still be big concerns.

I was aware, that's why I mentioned it as an example. I'm also aware you work for the company behind uv, which is why I wasn't too hopeful of this PR, but I wanted to raise my concerns anyway.

Yeah, although I want to disclaim that my decision to switch to the uv build backend was mine alone and not influenced by anything about my employer. I've been slowly migrating all of the projects I maintain to uv because I've found it easier to maintain development environments with.