Skip to content

Job/project persistence opens a fresh SQLite connection per write #32

Description

@bscholer

Problem

Every job event, every project history entry, every record update opens a fresh SQLite connection, runs migrate(conn) (idempotent but non-zero), writes one row, closes. See:

  • server/jobs.py:_persist_record (line 415)
  • server/jobs.py:_persist_event (line 469)
  • server/projects.py:_persist_project (line 444)
  • server/projects.py:_persist_history_entry (line 496)
  • server/projects.py:_persist_history_published (line 525)

server/catalog/db.py:connect() runs migrations on every connection, including the no-op version probe.

Why it bites

seq_register already fires node_progress per Siril log line (one connection per progress message). GraXpert and StarNet++ stream subprocess output line by line via server/subproc.py. As more streaming nodes ship, the worker thread will spend a meaningful fraction of its time on connect+migrate+commit cycles instead of doing work.

Today, fine. By Phase 3 / 4, painful.

Fix

Options, in order of effort:

  1. Cheap: keep one persistence connection per JobManager / ProjectManager, opened in __init__, reused for all writes. SQLite handles serial writes from a single connection without trouble; we already serialize through self._lock upstream.
  2. Medium: dedicated background writer thread with a queue.Queue of write commands; the worker just enqueues. Decouples Siril log throughput from the SQLite WAL fsync.
  3. Migration only at startup: connect() runs migrate() every time. Move migration to a one-shot at process start (lifespan in api.py); the per-write connect() should just open and set pragmas.

Even (1) and (3) together are mostly mechanical and would clear the bottleneck for now.

Verify

Bench a stack run with ~500 progress events and compare wall-clock + sqlite writes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    infraBuild, deploy, packagingtech debtStructural / correctness debt to pay down

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions