Skip to content

Migrate to Python 3.12/3.13, uv, and Keras 3 (.keras format)#1233

Open
DocGarbanzo wants to merge 18 commits intomainfrom
main-py312-upgrade
Open

Migrate to Python 3.12/3.13, uv, and Keras 3 (.keras format)#1233
DocGarbanzo wants to merge 18 commits intomainfrom
main-py312-upgrade

Conversation

@DocGarbanzo
Copy link
Copy Markdown
Contributor

@DocGarbanzo DocGarbanzo commented Apr 19, 2026

What this PR does

Migrates donkeycar from Python 3.11 + conda to Python 3.12/3.13 + uv,
upgrades TensorFlow to 2.19, and removes all Keras 2 / SavedModel legacy code
that broke under Keras 3.

Version bump: 5.3.x5.4.dev1 (breaking changes — see below).


Platform matrix

Platform Python TF Notes
Mac 3.12 2.19 + Metal tensorflow-metal==1.2.0 has no Python 3.13 wheel
PC / Linux 3.12 2.19 TF SavedModel export broken on Python 3.13
Raspberry Pi 3.13 — (TFLite only) Must match Debian system Python to reach libcamera/picamera2

Install instructions

1 — Install uv (all platforms)

curl -LsSf https://astral.sh/uv/install.sh | sh

2a — Raspberry Pi (Python 3.13, Debian Trixie / Pi OS Bookworm+)

Pi uses the Debian system Python 3.13 directly. Camera support
(libcamera, picamera2) is only available as Debian packages installed
into that system Python — they are not on PyPI. The venv must be created
with --system-site-packages so it can see them.

# Install camera system packages first
sudo apt install python3-libcamera python3-picamera2

# Create venv from the system Python 3.13 with access to system packages
uv venv ~/env --python 3.13 --system-site-packages

# Activate on login
echo 'source ~/env/bin/activate' >> ~/.bashrc
source ~/env/bin/activate

# User install (PyPI release):
uv pip install donkeycar[pi]

# Developer install (git clone):
uv pip install -e ".[pi,dev]"

Why not Python 3.12 on Pi?
python3-libcamera and python3-picamera2 are Debian packages installed
under /usr/lib/python3/dist-packages. A custom Python 3.12 venv's
--system-site-packages only reads /usr/local/lib/python3.12/site-packages
— it cannot see the Debian packages. Python 3.13 is the Debian system
Python, so its venv resolves both locations correctly.

Why no full TensorFlow on Pi?
Pi uses ai-edge-litert (Google's official TFLite successor, replacing the
dead tflite-runtime package). Inference-only; no training on Pi.


2b — Mac (Python 3.12)

# Create a persistent named venv
uv venv ~/.venvs/donkeycar --python 3.12

# Activate on login
echo 'source ~/.venvs/donkeycar/bin/activate' >> ~/.zshrc
source ~/.venvs/donkeycar/bin/activate

# User install (PyPI release):
uv pip install donkeycar[macos]

# Developer install (git clone):
uv pip install -e ".[macos,dev]"

Why Python 3.12, not 3.13?
tensorflow-metal==1.2.0 (required for GPU acceleration on Apple Silicon)
has no Python 3.13 wheel. TF 2.19 is the last version compatible with
tensorflow-metal==1.2.0. Additionally, TF's SavedModel export
(model.export()) is broken on Python 3.13 due to inspect.getattr_static
behaviour changes.


2c — PC / Linux (Python 3.12)

# Create a persistent named venv
uv venv ~/.venvs/donkeycar --python 3.12

# Activate on login (bash)
echo 'source ~/.venvs/donkeycar/bin/activate' >> ~/.bashrc
source ~/.venvs/donkeycar/bin/activate

# User install (PyPI release):
uv pip install donkeycar[pc]

# Developer install (git clone):
uv pip install -e ".[pc,dev]"

Breaking changes

Change Action required
Python 3.11 no longer supported Recreate your venv with Python 3.12 (Mac/PC) or 3.13 (Pi)
conda removed Uninstall old conda env; follow uv install steps above
.savedmodel models cannot be loaded Retrain your pilots — Keras 3 cannot load SavedModel directories. New models save as .keras automatically
picamera2 removed from pip extras Install via sudo apt install python3-picamera2 on Pi
tflite-runtime replaced by ai-edge-litert Run uv pip install donkeycar[pi] — handled automatically
RPi.GPIO replaced by gpiozero Run uv pip install donkeycar[pi] — handled automatically

Key code changes

  • pyproject.toml: replaces setup.cfg + MANIFEST.in; all metadata in one file
  • donkeycar/__init__.py: version 5.4.dev1; Python floor check updated to 3.12 (Mac/PC) / 3.13 (Pi)
  • parts/interpreter.py: removed broken TensorRT.load() branch that called keras.models.load_model() on a .savedmodel dir (not supported in Keras 3); added tensor-API fallback in TfLite interpreter for models without serving_default signature
  • pipeline/training.py: TRT export unconditionally uses model.export() (dead .savedmodel guard removed)
  • pipeline/database.py: .savedmodel removed from name-stripping and prefix map
  • management/ui/: all file filters and sync buttons updated from .savedmodel.keras
  • scripts/convert_to_tflite.py: accepts .keras input instead of .savedmodel
  • templates/complete.py, simulator.py: .savedmodel removed from model-path detection
  • parts/keras.py, pipeline/training.py, management/: tensorflow.python.keras.*tensorflow.keras.* (removed in TF 2.16)
  • .github/workflows/python-package.yml: replaced conda workflow; explicitly creates venv with matrix Python version before install

Test plan

  • pytest donkeycar/tests/test_keras.py — 8 pass, 1 xfail (KerasLSTM, CudnnRNNV3 unsupported in TFLite on macOS)
  • Train a model → verify saved as .keras
  • Load .keras model in UI (pilot screen, train screen transfer selector)
  • Sync screen shows buttons: h5, keras, tflite, trt
  • On Pi (Python 3.13 + --system-site-packages): camera works, TFLite inference runs, donkey drive completes

🤖 Generated with Claude Code

DocGarbanzo and others added 18 commits April 13, 2026 21:03
## Summary
- port the scoped Python 3.13 upgrade work onto a clean branch from
  `docgarbanzo/main`
- add a step-by-step branch plan in `MAIN_PY313_UPGRADE_PLAN.md`
- keep the detailed upgrade notes in `PYTHON313_UPGRADE.md`

## Dependency and CI updates
- require Python 3.13 in `setup.cfg`
- update TensorFlow / tf-keras to the Python 3.13-compatible 2.21 series
- switch Pi TFLite runtime dependency to `ai-edge-litert`
- update the conda CI workflow to test Python 3.13

## TensorFlow / Keras compatibility
- replace deprecated `tensorflow.python.*` imports with supported
  `tensorflow.keras.*` and `tf.saved_model.*` APIs
- update Keras 3 model save/load expectations and `.keras` handling
- adjust Keras training and shape handling for current TensorFlow/Keras
  behavior
- make TensorFlow-dependent code import-safe when TensorFlow is absent

## LiteRT / Pi inference compatibility
- add TFLite tensor-API fallback for models without signatures
- keep explicit interpreter resolution for `ai-edge-litert` vs full
  TensorFlow installs
- make `output_shapes()` work without TensorFlow installed
- harden shape handling for scalar and NumPy integer inputs used during
  LiteRT inference

## Tests
- skip TensorFlow-dependent tests when TensorFlow is not installed
- update script and training tests for Keras 3 behavior
- run the full suite in the `donkey313` environment

## Validation
- `pytest` on `main-py313-upgrade` in `donkey313`
  - `163 passed, 16 skipped, 1 xfailed`

## Notes
- this branch was rebuilt from synced `docgarbanzo/main` and only carries
  the scoped Python 3.13 / LiteRT upgrade work, not unrelated `dev`
  changes from `python313-upgrade`
tensorflow-metal has no wheel for Python 3.13, causing installation to
fail on macOS. Removing it until Apple ships a compatible release; TF
2.21 will run CPU-only on macOS under Python 3.13.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t.toml

- Replace conda with uv for environment management; document venv locations
  (Pi: ~/env, Mac/PC: ~/.venvs/donkeycar) and activation in shell profile
- Migrate all setup.cfg and MANIFEST.in content into pyproject.toml; delete
  both files so pyproject.toml is the single source of package metadata
- Pin TF to 2.19.* across pc and macos extras — highest version compatible
  with tensorflow-metal 1.2.0 (TF 2.20+ breaks Metal GPU on macOS)
- Add tensorflow-metal==1.2.0 to macos extra (first release with cp312 wheel)
- Replace conda-based CI with uv + astral-sh/setup-uv@v5 on Python 3.12;
  rename workflow file to python-package.yml
- Update Makefile: uv build and uv run pytest
- Update README: install section with uv workflow; fix CI badge URL
- Consolidate four migration/upgrade planning docs into PYTHON312_MIGRATION.md;
  delete PYTHON312_SYSTEM_INSTALL.md, PYTHON313_UPGRADE.md,
  MAIN_PY313_UPGRADE_PLAN.md, UV_MIGRATION_PLAN.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
picamera2 depends on libcamera which is only available as a system
package (python3-libcamera). Installing via pip causes ModuleNotFoundError
at runtime. The venv must be created with --system-site-packages.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On Raspberry Pi OS Trixie the system Python is 3.13 and picamera2/libcamera
are Debian packages (not pip-installable). The venv must use Python 3.13 with
--system-site-packages to access them. Updated requires-python to <3.14,
added Python 3.13 classifier, and documented the correct Pi setup in README.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pi uses system Python 3.13 (camera libraries are Debian packages, not pip).
Mac/PC must stay on Python 3.12 (TF 2.19 + tensorflow-metal broken on 3.13).
Corrects stale claims about uv bundled CPython on Pi and updates install
workflow, dependency table, and document title accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Python dependency table: split Python row into Mac/PC (3.12) and Pi (3.13)
- Install workflow: add echo activation line to Mac/PC blocks; remove
  duplicate from First-time setup section
- Docs update table: note Pi uses Python 3.13, not 3.12

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keras 3 (TF 2.16+) no longer supports saving/loading Keras models as
SavedModel directories via keras.models.load_model(). The native format
is now .keras. SavedModel is retained only as an internal intermediate
for TensorRT conversion.

Changes:
- Replace .savedmodel with .keras in all UI filters and transfer logic
- Remove dead guard in training.py that checked for .savedmodel input
- Update KerasInterpreter.load() comment to reflect export-time losses
- Remove broken TensorRT.load() branch that tried to load .savedmodel
  as a Keras model (broken in Keras 3; users should use .trt instead)
- Update test variable names for clarity

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Version bump reflects breaking changes: Python 3.12 now required on
PC/Mac, 3.13+ on RPi. Update startup message to clarify platform
requirements.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The nano extra pinned numpy==1.23.* which conflicts with the project's
numpy>=1.26.0 base dependency. Jetson Nano hardware cannot run Python
3.12, so the extra is dead code on this branch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv run without --no-sync validates all extras for compatibility before
running, causing a conflict between nano's numpy==1.23.* and the base
numpy>=1.26.0. Since the venv is already set up by the preceding
uv pip install step, re-syncing is unnecessary.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- KerasLSTM: mark xfail (CudnnRNNV3 op not supported in TFLite without CUDA)
- Keras3D_CNN: remove xfail (Flex delegate now auto-loaded via ai-edge-litert)
- CI: explicitly create venv with matrix Python version before installing,
  so tests always run on the correct Python rather than the system default

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CudnnRNNV3 is unsupported in TFLite on macOS but works on Linux; using
strict=False allows XPASS on Linux and XFAIL on macOS without failing CI.
Also add .venv to .gitignore to prevent accidental project-local venv commits.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant