Skip to content

Add persistent observability history tracking#438

Open
alex-bea wants to merge 1 commit intomainfrom
codex/introduce-persistent-store-for-observability-history
Open

Add persistent observability history tracking#438
alex-bea wants to merge 1 commit intomainfrom
codex/introduce-persistent-store-for-observability-history

Conversation

@alex-bea
Copy link
Owner

@alex-bea alex-bea commented Nov 11, 2025

Summary

  • persist DIS observability reports via a JSON-backed history index and reload the cache on collector startup
  • enrich observability reports with alert routing decisions and summary severity metadata
  • add restart coverage to verify history availability and blocking behaviour for critical alerts

Testing

  • pytest tests/observability/test_dis_observability_history.py

Codex Task


Note

Persist observability reports to a JSON history (reloaded on startup) and add alert routing/severity metadata with paging/blocking, plus tests.

  • Observability Core:
    • Add AlertDecision dataclass and compute report.alert_decision (severity, paging, block, triggers).
    • Implement alert dispatching via logger based on decision.
  • Persistence:
    • Introduce ObservabilityHistoryStore to append/load observability_history.json.
    • Collector now loads history on init, persists each report (with serialized payload), and retains last_alert_decision.
    • Add custom dataclass serialization and hydration for reports.
  • API & Summary:
    • Export AlertDecision in observability/__init__.py.
    • get_observability_summary now includes alert_severity and block_run per dataset.
  • Tests:
    • Add tests/observability/test_dis_observability_history.py covering history persistence across restarts and critical alert blocking behavior.

Written by Cursor Bugbot for commit 613e20c. This will update automatically on new commits. Configure here.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on December 10

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

warnings=data.get("warnings", []),
)

return report
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: History Reports: Time Rewrites The Past

When loading reports from history, _report_from_dict reconstructs metric dataclasses which triggers their __post_init__ methods. These methods recalculate time-dependent values like staleness_hours using datetime.utcnow() instead of the original report timestamp, causing historical "healthy" reports to appear increasingly stale over time. The recalculated alerts then cause DISObservabilityReport.__post_init__ to generate incorrect alert_decision values, breaking the persistence feature where a "normal" severity report could become "critical" when reloaded.

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant