Skip to content

Add chunked sync for large .archcore/ trees #25

Description

@ivklgn

Problem

The sync payload builder sends everything in one request (internal/sync/payload.go), which won't scale to very large .archcore/ trees (10k+ files).

Expected result

Sync batches large payloads into chunks, preserving the "update the manifest only after a confirmed server response" invariant per batch.

Impact

Lets very large knowledge bases sync without exceeding request limits.

Scope

  • Batch the payload above a threshold.
  • Preserve the manifest-only-on-success invariant per batch.
  • Tests with a synthetic large corpus.

Files: internal/sync/payload.go, internal/sync/manifest.go, cmd/sync.go (+ tests)

Acceptance: a 10k+-file .archcore/ syncs in batches without exceeding request limits.

Status: blocked — the sync feature is currently gated off, and this targets a scale not yet observed in practice.

Metadata

Metadata

Assignees

No one assigned

    Labels

    blockedDepends on out-of-repo or gated workenhancementNew feature or requestperformanceRead-path and scaling

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions