Migrate to Python 3.12/3.13, uv, and Keras 3 (.keras format)#1233
Open
DocGarbanzo wants to merge 18 commits intomainfrom
Open
Migrate to Python 3.12/3.13, uv, and Keras 3 (.keras format)#1233DocGarbanzo wants to merge 18 commits intomainfrom
DocGarbanzo wants to merge 18 commits intomainfrom
Conversation
## Summary - port the scoped Python 3.13 upgrade work onto a clean branch from `docgarbanzo/main` - add a step-by-step branch plan in `MAIN_PY313_UPGRADE_PLAN.md` - keep the detailed upgrade notes in `PYTHON313_UPGRADE.md` ## Dependency and CI updates - require Python 3.13 in `setup.cfg` - update TensorFlow / tf-keras to the Python 3.13-compatible 2.21 series - switch Pi TFLite runtime dependency to `ai-edge-litert` - update the conda CI workflow to test Python 3.13 ## TensorFlow / Keras compatibility - replace deprecated `tensorflow.python.*` imports with supported `tensorflow.keras.*` and `tf.saved_model.*` APIs - update Keras 3 model save/load expectations and `.keras` handling - adjust Keras training and shape handling for current TensorFlow/Keras behavior - make TensorFlow-dependent code import-safe when TensorFlow is absent ## LiteRT / Pi inference compatibility - add TFLite tensor-API fallback for models without signatures - keep explicit interpreter resolution for `ai-edge-litert` vs full TensorFlow installs - make `output_shapes()` work without TensorFlow installed - harden shape handling for scalar and NumPy integer inputs used during LiteRT inference ## Tests - skip TensorFlow-dependent tests when TensorFlow is not installed - update script and training tests for Keras 3 behavior - run the full suite in the `donkey313` environment ## Validation - `pytest` on `main-py313-upgrade` in `donkey313` - `163 passed, 16 skipped, 1 xfailed` ## Notes - this branch was rebuilt from synced `docgarbanzo/main` and only carries the scoped Python 3.13 / LiteRT upgrade work, not unrelated `dev` changes from `python313-upgrade`
tensorflow-metal has no wheel for Python 3.13, causing installation to fail on macOS. Removing it until Apple ships a compatible release; TF 2.21 will run CPU-only on macOS under Python 3.13. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t.toml - Replace conda with uv for environment management; document venv locations (Pi: ~/env, Mac/PC: ~/.venvs/donkeycar) and activation in shell profile - Migrate all setup.cfg and MANIFEST.in content into pyproject.toml; delete both files so pyproject.toml is the single source of package metadata - Pin TF to 2.19.* across pc and macos extras — highest version compatible with tensorflow-metal 1.2.0 (TF 2.20+ breaks Metal GPU on macOS) - Add tensorflow-metal==1.2.0 to macos extra (first release with cp312 wheel) - Replace conda-based CI with uv + astral-sh/setup-uv@v5 on Python 3.12; rename workflow file to python-package.yml - Update Makefile: uv build and uv run pytest - Update README: install section with uv workflow; fix CI badge URL - Consolidate four migration/upgrade planning docs into PYTHON312_MIGRATION.md; delete PYTHON312_SYSTEM_INSTALL.md, PYTHON313_UPGRADE.md, MAIN_PY313_UPGRADE_PLAN.md, UV_MIGRATION_PLAN.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
picamera2 depends on libcamera which is only available as a system package (python3-libcamera). Installing via pip causes ModuleNotFoundError at runtime. The venv must be created with --system-site-packages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On Raspberry Pi OS Trixie the system Python is 3.13 and picamera2/libcamera are Debian packages (not pip-installable). The venv must use Python 3.13 with --system-site-packages to access them. Updated requires-python to <3.14, added Python 3.13 classifier, and documented the correct Pi setup in README. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pi uses system Python 3.13 (camera libraries are Debian packages, not pip). Mac/PC must stay on Python 3.12 (TF 2.19 + tensorflow-metal broken on 3.13). Corrects stale claims about uv bundled CPython on Pi and updates install workflow, dependency table, and document title accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Python dependency table: split Python row into Mac/PC (3.12) and Pi (3.13) - Install workflow: add echo activation line to Mac/PC blocks; remove duplicate from First-time setup section - Docs update table: note Pi uses Python 3.13, not 3.12 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keras 3 (TF 2.16+) no longer supports saving/loading Keras models as SavedModel directories via keras.models.load_model(). The native format is now .keras. SavedModel is retained only as an internal intermediate for TensorRT conversion. Changes: - Replace .savedmodel with .keras in all UI filters and transfer logic - Remove dead guard in training.py that checked for .savedmodel input - Update KerasInterpreter.load() comment to reflect export-time losses - Remove broken TensorRT.load() branch that tried to load .savedmodel as a Keras model (broken in Keras 3; users should use .trt instead) - Update test variable names for clarity Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Version bump reflects breaking changes: Python 3.12 now required on PC/Mac, 3.13+ on RPi. Update startup message to clarify platform requirements. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The nano extra pinned numpy==1.23.* which conflicts with the project's numpy>=1.26.0 base dependency. Jetson Nano hardware cannot run Python 3.12, so the extra is dead code on this branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
uv run without --no-sync validates all extras for compatibility before running, causing a conflict between nano's numpy==1.23.* and the base numpy>=1.26.0. Since the venv is already set up by the preceding uv pip install step, re-syncing is unnecessary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- KerasLSTM: mark xfail (CudnnRNNV3 op not supported in TFLite without CUDA) - Keras3D_CNN: remove xfail (Flex delegate now auto-loaded via ai-edge-litert) - CI: explicitly create venv with matrix Python version before installing, so tests always run on the correct Python rather than the system default Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CudnnRNNV3 is unsupported in TFLite on macOS but works on Linux; using strict=False allows XPASS on Linux and XFAIL on macOS without failing CI. Also add .venv to .gitignore to prevent accidental project-local venv commits. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Migrates donkeycar from Python 3.11 + conda to Python 3.12/3.13 + uv,
upgrades TensorFlow to 2.19, and removes all Keras 2 / SavedModel legacy code
that broke under Keras 3.
Version bump:
5.3.x→5.4.dev1(breaking changes — see below).Platform matrix
tensorflow-metal==1.2.0has no Python 3.13 wheellibcamera/picamera2Install instructions
1 — Install uv (all platforms)
curl -LsSf https://astral.sh/uv/install.sh | sh2a — Raspberry Pi (Python 3.13, Debian Trixie / Pi OS Bookworm+)
Pi uses the Debian system Python 3.13 directly. Camera support
(
libcamera,picamera2) is only available as Debian packages installedinto that system Python — they are not on PyPI. The venv must be created
with
--system-site-packagesso it can see them.2b — Mac (Python 3.12)
2c — PC / Linux (Python 3.12)
Breaking changes
.savedmodelmodels cannot be loaded.kerasautomaticallypicamera2removed from pip extrassudo apt install python3-picamera2on Pitflite-runtimereplaced byai-edge-litertuv pip install donkeycar[pi]— handled automaticallyRPi.GPIOreplaced bygpiozerouv pip install donkeycar[pi]— handled automaticallyKey code changes
pyproject.toml: replacessetup.cfg+MANIFEST.in; all metadata in one filedonkeycar/__init__.py: version5.4.dev1; Python floor check updated to 3.12 (Mac/PC) / 3.13 (Pi)parts/interpreter.py: removed brokenTensorRT.load()branch that calledkeras.models.load_model()on a.savedmodeldir (not supported in Keras 3); added tensor-API fallback inTfLiteinterpreter for models withoutserving_defaultsignaturepipeline/training.py: TRT export unconditionally usesmodel.export()(dead.savedmodelguard removed)pipeline/database.py:.savedmodelremoved from name-stripping and prefix mapmanagement/ui/: all file filters and sync buttons updated from.savedmodel→.kerasscripts/convert_to_tflite.py: accepts.kerasinput instead of.savedmodeltemplates/complete.py,simulator.py:.savedmodelremoved from model-path detectionparts/keras.py,pipeline/training.py,management/:tensorflow.python.keras.*→tensorflow.keras.*(removed in TF 2.16).github/workflows/python-package.yml: replaced conda workflow; explicitly creates venv with matrix Python version before installTest plan
pytest donkeycar/tests/test_keras.py— 8 pass, 1 xfail (KerasLSTM, CudnnRNNV3 unsupported in TFLite on macOS).keras.kerasmodel in UI (pilot screen, train screen transfer selector)h5,keras,tflite,trt--system-site-packages): camera works, TFLite inference runs,donkey drivecompletes🤖 Generated with Claude Code