Skip to content

Conversation

@newgene
Copy link
Member

@newgene newgene commented Dec 23, 2025

When we test the latest Python 3.14 (both regular and free-threaded builds) with the biothings hub at 1.1.x branch, we had this error for any manifest-based data plugin during either dumping or uploading steps:

Full traceback in Python 3.14
Process ForkServerProcess-2:
Traceback (most recent call last):
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module 'biothings.hub.dataplugin.templates' has no attribute 'AssistedDumper_hpo'. Did you mean: 'AssistedDumper'?
Process ForkServerProcess-1:
Traceback (most recent call last):
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/multiprocessing/queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module 'biothings.hub.dataplugin.templates' has no attribute 'AssistedDumper_hpo'. Did you mean: 'AssistedDumper'?
INFO:tornado.access:200 GET /job_manager (172.24.240.102) 9.06ms
ERROR:asyncio:Exception in callback JobManager.defer_to_process.<locals>.run.<locals>.ran() at /home/biothings/biothings_studio/biothings/utils/manager.py:486
handle: <Handle JobManager.defer_to_process.<locals>.run.<locals>.ran() at /home/biothings/biothings_studio/biothings/utils/manager.py:486>
Traceback (most recent call last):
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/events.py", line 94, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/biothings/biothings_studio/biothings/utils/manager.py", line 490, in ran
    f.result()
    ~~~~~~~~^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR:asyncio:Exception in callback JobManager.defer_to_process.<locals>.run.<locals>.ran() at /home/biothings/biothings_studio/biothings/utils/manager.py:486
handle: <Handle JobManager.defer_to_process.<locals>.run.<locals>.ran() at /home/biothings/biothings_studio/biothings/utils/manager.py:486>
Traceback (most recent call last):
  File "/home/biothings/.local/share/uv/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/asyncio/events.py", line 94, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/biothings/biothings_studio/biothings/utils/manager.py", line 490, in ran
    f.result()
    ~~~~~~~~^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR:dump_hpo:Error downloading 'http://purl.obolibrary.org/obo/hp/hpoa/phenotype_to_genes.txt': A process in the process pool was terminated abruptly while the future was running or pending.
Traceback (most recent call last):
  File "/home/biothings/biothings_studio/biothings/hub/dataload/dumper.py", line 616, in done
    _ = f.result()
  File "/home/biothings/biothings_studio/biothings/utils/manager.py", line 501, in run
    res = await res
          ^^^^^^^^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR:dump_hpo:Error downloading 'http://purl.obolibrary.org/obo/hp/hpoa/phenotype_to_genes.txt': A process in the process pool was terminated abruptly while the future was running or pending.
Traceback (most recent call last):
  File "/home/biothings/biothings_studio/biothings/hub/dataload/dumper.py", line 616, in done
    _ = f.result()
  File "/home/biothings/biothings_studio/biothings/utils/manager.py", line 501, in run
    res = await res
          ^^^^^^^^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR:dump_hpo:Error while dumping source: A process in the process pool was terminated abruptly while the future was running or pending.
ERROR:dump_hpo:Traceback (most recent call last):
  File "/home/biothings/biothings_studio/biothings/hub/dataload/dumper.py", line 467, in dump
    await self.do_dump(job_manager=job_manager)
  File "/home/biothings/biothings_studio/biothings/hub/dataload/dumper.py", line 643, in do_dump
    await asyncio.gather(*jobs)
  File "/home/biothings/biothings_studio/biothings/hub/dataload/dumper.py", line 616, in done
    _ = f.result()
  File "/home/biothings/biothings_studio/biothings/utils/manager.py", line 501, in run
    res = await res
          ^^^^^^^^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

ERROR:dump_hpo:failed [steps=dump,post]: A process in the process pool was terminated abruptly while the future was running or pending.

The fix for now is to force the fork behavior of the multiprocessing module in Python 3.14, but eventually we should consider refactor the code to work with the "safer" forkserver mode.

REF: https://docs.python.org/3.14/library/multiprocessing.html#contexts-and-start-methods

@newgene newgene merged commit 48b3697 into 1.1.x Dec 24, 2025
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants