[RFC] stop using conda for Python CI jobs

[jameslamb]

Proposal

We should stop using conda + conda-forge for Python-package CI and instead do something like:

• linting, docs, etc. --> pixi
• Python unit tests for end-of-life versions --> pixi
• all other Python build and unit test jobs --> pip

Motivation

This project uses conda + the conda-forge channel to set up environments for building and testing Python wheels.

LightGBM/.ci/test.sh

Lines 114 to 120 in cf16013

	CONDA_REQUIREMENT_FILE="${BUILD_DIRECTORY}/.ci/conda-envs/ci-core.txt"
	conda create \
	-y \
	-n "${CONDA_ENV}" \
	--file "${CONDA_REQUIREMENT_FILE}" \
	"${CONDA_PYTHON_REQUIREMENT}" \
	|| exit 1

That has been helpful in many ways here, especially:

• strong runtime compatibility guarantees across all packages (GLIBC version, supported Python versions, etc.)
• broad platform support (including PowerPC by default, cross-ref [ci] add ppc64le C++ test job #7345)
• ease of local development (in some ways)

But that setup has some significant drawbacks:

• elevated risk of building DSOs (.dll / .dylib / .so) that depend on conda-specific details by using conda-forge's compilation toolchain and shared libraries
  • details: [ci] Remove gcc tests on MacOS #6566 (review)
• elevated risk of failures from mixing conda, pip, and system-installed libraries
  • example: Python package looks for library in wrong path #5106 (comment)
  • another example: [ci] avoid overwriting installed 'build' in build-python.sh #7356
• CI fails to thoroughly cover the most common (I'm guessing) way lightgbm is used... by installing wheels into an environment managed with pip / uv
  • e.g., miss out on understanding which system libraries lightgbm and its dependencies expect to be installed
  • e.g., potentially miss issues from mixing manylinux wheels vendoring the same stuff, like Feature Request: add timeout parameter to the .fit() method  #6596

In my opinion, those drawbacks significantly outweigh the benefits.

wheels, not conda packages, are built from this repo... they should be built and tested in environments built by installing wheels

Benefits of this work

See "Motivation" above, but in short:

• makes CI more closely match a very common workflow for users
• simplifies some details of building Python wheels (which may help with [cmake] Simplify and improve CMake configuration #6774 and [RFC] [CUDA] distribute Python wheels with CUDA support #6828, among others)

Approach

It'd be convenient to get the CI image building finally moved into this repo (#7011, work in progress in #7109) before making other changes.

Then, remove conda from relevant places in .ci/ scripts, and do whatever's needed to get it passing again installing libraries from some mix of system package managers and pip.

Notes

Based on this conversation @borchero and I had recently: #7328 (comment)