Skip to content

A1 — Remove downstream source ceilings (uncapped per-topic ingestion) #8

Description

@gnanirahulnutakki

Part of the content-flow redesign — see design doc ardur-pipeline/docs/redesign-content-flow-2026-06.md (ArdurAI/ardur-pipeline#14).

A1 — Remove downstream source ceilings (uncapped per-topic ingestion)

The aggregator already has no upper cap (diversityFloor: 20 in src/sources.ts is a lower bound). The effective ceilings live downstream and must be removed so the full clustered set survives to synthesis:

  • ardur-top10-engine Top10Entry.references capped at 5 (handled in T1).
  • ardur-article-synthesizer references.slice(0, 10) (handled in S1).

Scope (this repo)

  • Confirm ingestion keeps all deduped members per cluster (no silent truncation in src/index.ts cluster assembly).
  • Carry the full member/source set forward in AggregationData (works with rev-3 documentsByTopic).
  • Practical limits only: per-source fetch budget, dedup, paywall/ToS. Log any drop with a reason (no silent caps).

Acceptance

  • A topic with >30 clustered sources emits >30 members; no constant/slice caps the count.
  • Coverage report (coverageByTopic) reflects the true ingested/deduped counts.

Depends on: C1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requeststatus:readyScoped, ready for implementation (Codex can pick up)

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions