RFC: Minimum Linux distribution version support policy

[hawkinsp]

Currently JAX tags its jaxlib wheels as manylinux_2_27, which declares compatibility with Ubuntu 18.04. This is a very old Linux distribution, so old that it limits our ability to use recent versions of C++ (e.g., C++20), and that is making our engineers sad.

We are tentatively proposing to use the following rule: we will support each Ubuntu LTS release for 5 years after its initial release, which corresponds to Ubuntu's standard support period for that distribution. For a table translating that to other Linux distributions, see
https://github.com/mayeut/pep600_compliance?tab=readme-ov-file#distro-compatibility

i.e. the policy would be:

• right now, we will update our minimum Linux version to Ubuntu 22.04 aka manylinux_2_35.
• in April 2027, we will update our minimum Linux version to Ubuntu 24.04, aka manylinux_2_39.
• in April 2029, we will update our minimum Linux version to Ubuntu 26.04, whose manylinux tag is to be determined.

It is possible that a 5 year support period is too short. Please let us know if you feel that way.

Comments? Feedback?

[PhilipVinc]

This would wreak havoc on poor public researchers. :(

For reasons I cannot understand, HPC administrators love to use very old distros even in recent machines.
I think the arguments of the IT that handle them is that this way there 'are less compatibility problems'. Which is nonsense, but... that's what we have to live with. And it's hard to convince them of anything.

France's 'newest' supercomputer (Jean Zay, with 2k A100s and 1k H100s), which was last 'upgraded' 1 year ago, has the following ldd/GLIBC

[udb21rp@jean-zay1: ~]$ LIBC=$(ldconfig -p | awk '/libc\.so\.6/ {print $NF; exit}')
[udb21rp@jean-zay1: ~]$ strings "$LIBC" | grep -o 'GLIBC_[0-9.]\+' | sort -Vu | tail   # newest exported
GLIBC_2.32
GLIBC_2.33
GLIBC_2.34
GLIBC_2.35

I know for sure the next maintenance is programmed for next summer, but they rarely update ldd...

Leonardo, one of the largest EU HPC centers with several thousands A100s and widely used, has the most recent GLIBC at GLIBC_2.28.
A locally-administered cluster with A100s we have in my institution supports GLIBC_2.25 and can't run recent jax anymore.

Moreover, as those are slurm managed, we cannot use containers.

Ask any academic who can ssh on an HPC cluster and they will tell you that the glib versions supported is really a problem...

[PhilipVinc]

(is a MUSL version of jaxlib an impossibility?)

[hawkinsp]

This was the more aggressive of the two proposals we were thinking of making.

The less aggressive one was a 7 year time horizon. Under that proposal:

• right now, the minimum becomes Ubuntu 20.04 aka manylinux_2_31 (corresponding to us dropping Ubuntu 18.04 after 7 years)
• in April 2027, the minimum becomes Ubuntu 22.04, aka manylinux_2_35.
• In April 2029, the minimum becomes Ubuntu 24.04, aka manylinux_2_39.

Would that be more acceptable? It seems that would be ok for Zay (as would the previous plan), but not Leonardo.

Supporting older glibc versions is a large amount of work since we essentially have to build custom toolchains that can target them: we don't want to be limited to language features from over 7 years ago.

Further, the point of this discussion is to set a policy for the future so we don't need to guess when it is safe to upgrade.

WDYT?

[PhilipVinc]

Would this change be in XLA, so it would affect Tensorflow as well?

European HPC centers are very... bureaucratic, but if google was to set out a white paper/roadmap paper (even just 1 page long) arguing why they cannot support older GLIBC, and that you will in the next ~12 months fading out support, it would make things easier for me/us to make a case with the IT administration and say 'that's what we need.
AI is a priority, so they do listen. It's just that they don't listen to the few 'vocal' users. But if YOU write something, they will give it a bit more weight.

I would be happy to help you do it if I can be of any assistance, and then forward it to the relevant administrations.
(Seriously, this would really be a huge assistance for us in academic HPC. If you want feel free to contact me directly at my email address on my GitHub username).

--

A public roadmap would help, because we could trigger IT centers in advance. Roadmaps are long-term (~12 months minimum) so that is the minimum warning time that would allow us not to suffer too much.

If you start to release a roadmap for future deprecations, even if it accelerates the pace in the future, the fact that it's public might allow us to negotiate with HPC centers.

But to be effective, we'd need something more (a whitepaper/position paper). Every time I ask HPC centers to support recent software, they complain with us users that 'want the bleeding edge jax', and they don't listen to the fact that we are actively getting bugs fixed in the last releases...