Problem
Every job event, every project history entry, every record update opens a fresh SQLite connection, runs migrate(conn) (idempotent but non-zero), writes one row, closes. See:
server/jobs.py:_persist_record (line 415)
server/jobs.py:_persist_event (line 469)
server/projects.py:_persist_project (line 444)
server/projects.py:_persist_history_entry (line 496)
server/projects.py:_persist_history_published (line 525)
server/catalog/db.py:connect() runs migrations on every connection, including the no-op version probe.
Why it bites
seq_register already fires node_progress per Siril log line (one connection per progress message). GraXpert and StarNet++ stream subprocess output line by line via server/subproc.py. As more streaming nodes ship, the worker thread will spend a meaningful fraction of its time on connect+migrate+commit cycles instead of doing work.
Today, fine. By Phase 3 / 4, painful.
Fix
Options, in order of effort:
- Cheap: keep one persistence connection per
JobManager / ProjectManager, opened in __init__, reused for all writes. SQLite handles serial writes from a single connection without trouble; we already serialize through self._lock upstream.
- Medium: dedicated background writer thread with a
queue.Queue of write commands; the worker just enqueues. Decouples Siril log throughput from the SQLite WAL fsync.
- Migration only at startup:
connect() runs migrate() every time. Move migration to a one-shot at process start (lifespan in api.py); the per-write connect() should just open and set pragmas.
Even (1) and (3) together are mostly mechanical and would clear the bottleneck for now.
Verify
Bench a stack run with ~500 progress events and compare wall-clock + sqlite writes.
Problem
Every job event, every project history entry, every record update opens a fresh SQLite connection, runs
migrate(conn)(idempotent but non-zero), writes one row, closes. See:server/jobs.py:_persist_record(line 415)server/jobs.py:_persist_event(line 469)server/projects.py:_persist_project(line 444)server/projects.py:_persist_history_entry(line 496)server/projects.py:_persist_history_published(line 525)server/catalog/db.py:connect()runs migrations on every connection, including the no-op version probe.Why it bites
seq_registeralready firesnode_progressper Siril log line (one connection per progress message). GraXpert and StarNet++ stream subprocess output line by line viaserver/subproc.py. As more streaming nodes ship, the worker thread will spend a meaningful fraction of its time on connect+migrate+commit cycles instead of doing work.Today, fine. By Phase 3 / 4, painful.
Fix
Options, in order of effort:
JobManager/ProjectManager, opened in__init__, reused for all writes. SQLite handles serial writes from a single connection without trouble; we already serialize throughself._lockupstream.queue.Queueof write commands; the worker just enqueues. Decouples Siril log throughput from the SQLite WAL fsync.connect()runsmigrate()every time. Move migration to a one-shot at process start (lifespanin api.py); the per-writeconnect()should just open and set pragmas.Even (1) and (3) together are mostly mechanical and would clear the bottleneck for now.
Verify
Bench a stack run with ~500 progress events and compare wall-clock + sqlite writes.