Fix multiprocessing for python 3.14#1127
Merged
xylar merged 9 commits intoMPAS-Dev:developfrom Dec 9, 2025
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request enhances the codebase's compatibility with Python 3.14 and newer versions by addressing multiprocessing serialization issues and modernizing resource handling patterns.
Key Changes:
- Implemented a clear/reload pattern for unpicklable file-based objects (namelists and streams) in AnalysisTask to support multiprocessing
- Dynamically detect Python version in test suite scripts instead of hardcoding version numbers
- Migrated from deprecated
importlib.resources.contents()toimportlib.resources.files().iterdir()API
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
mpas_analysis/shared/analysis_task.py |
Adds multiprocessing support by implementing __getstate__, start() override, and helper methods to clear/reload unpicklable attributes (namelists, streams) |
suite/run_dev_suite.bash |
Dynamically detects Python version using sys.version_info instead of hardcoded value |
suite/run_e3sm_unified_suite.bash |
Dynamically detects Python version using sys.version_info instead of hardcoded value |
mpas_analysis/__main__.py |
Updates to modern importlib.resources API for listing machine configuration files |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Collaborator
Author
|
It looks like we're hitting networkx/networkx#8362, which is caused by python/cpython#142214 |
Merged
7 tasks
xylar
force-pushed
the
fix-multiprocessing-for-python-3.14
branch
from
d9c2e87 to
14a9870
Compare
This was referenced Dec 4, 2025
xylar
force-pushed
the
fix-multiprocessing-for-python-3.14
branch
from
14a9870 to
9eef711
Compare
This reverts commit 10dd90f.
This reverts commit f7ad851.
Namelists and streams objects cannot be pickled, but multiprocessing in python 3.14 tries to pickle AnalysisTask objects. With this fix, the attributes associated with namelists and streams are cleared after setup_and_check() is called, and restored before run_task() is called.
Previously, the python version was hard-coded
We need to do this before starting the task, rather than after setup_and_check() because other tasks may try to access the namelists and streams from a parent or prerequisite task.
We don't want to rely on retrieving them from namelists at runtime because the `self.namelist` object may not exist. The mesh filename also needs to be stored since it isn't necessarily available from the streams object at runtime.
xylar
force-pushed
the
fix-multiprocessing-for-python-3.14
branch
from
9eef711 to
dbbe7ac
Compare
Collaborator
Author
TestingBoth CI and the test suite now runs successfully with python 3.14. For the latter, see: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces important improvements to the handling of unpicklable attributes and Python version detection in both the analysis task logic and suite scripts. The changes enhance compatibility with newer Python versions and multiprocessing, making the codebase more robust and maintainable.
Multiprocessing and Pickling Improvements
AnalysisTaskinmpas_analysis/shared/analysis_task.pyto clear and reload namelist and streams attributes before and after multiprocessing to avoid pickling issues. Added_clear_namelists_and_streamsand_load_namelists_and_streamshelper methods, updatedstartandrunmethods, and implemented a custom__getstate__to remove unpicklable attributes. [1] [2] [3] [4]Python Version Detection
suite/run_dev_suite.bashandsuite/run_e3sm_unified_suite.bashto dynamically detect the Python version usingsys.version_infoinstead of hardcoding it, improving compatibility with future Python releases. [1] [2] [3]Resource Handling Modernization
mpas_analysis/__main__.pyby replacing deprecatedimportlib.resources.contentswithimportlib.resources.filesand updating related logic for listing machine configuration files. [1] [2]Checklist
Testingcomment in the PR documents testing used to verify the changes