Skip to content

feat: Metrics tracking system for entanglement management #376

Open
Tw1ZZLER wants to merge 83 commits into
masterfrom
metrics
Open

feat: Metrics tracking system for entanglement management #376
Tw1ZZLER wants to merge 83 commits into
masterfrom
metrics

Conversation

@Tw1ZZLER

@Tw1ZZLER Tw1ZZLER commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Continuation of implementation of #368 and direct continuation of development after #374.

This PR implements a more generalized central metrics tracking system for SeQUeNCe. The module has been split up until multiple modules:

  • metrics/__init__.py: The front-facing API methods that are used to interact with the metrics module. Includes enable, record, configure, collect_trial_metrics, and aggregate_trial_metrics.
  • event_types.py: Defines methods for creating and handling event types. Event types are what is used by the record method to record a metric.
  • builtins.py: Built-in event types and metrics. Used for abstraction and categorizing the different types of metrics used throughout SeQUeNCe. Also gives an example of how a user could create their own metrics for their own experiments!
  • metrics.py: Mostly defines a Metric abstract class and the types of Metrics that are available. Each metric includes collect method that collects the information from those metrics, and this is typically used with the collect_trial_metrics method.
  • registry.py: Metric registry that provides convenient methods for registering and creating new metrics. All Metric types are stored as a list _metrics in this module.
  • storage.py: Mostly unchanged from MVP implementation of centralized metrics tracking system #374, just all storage methods have been moved to their own module, along with the the time provider classes.

Metric Superclass

Metrics have been generalized into certain metric types, that change the behavior of certain methods. Metric is now an abstract superclass of all the other metric types, which are implemented as data classes.

Abstract Properties

Metrics requires the abstract properties event_types and output_keys to be set:

  • event_types is an immutable set of the EventTypes that the metric reacts to during record.
  • output_keys is an immutable set of the keys that the collect() method produces in it's output dictionary. Useful for aggregation of data. collect() is discussed in the next section.

Collect Method

Metric also requires an abstract method in the form of collect(). The collect() method is to generalize the global collect_trial_metrics() method, which now just calls the collect() method of each metric and puts them all into one list. collect() is implemented differently and outputs different sets of information based on the metric.

  • CounterPairMetric collects the failures, successes, and success rate of the passed owner. This helps generalize the successes and failures that all the entanglement protocols produce.
  • LastValueMetric collects the last scalar field value from matching events. This helps generalize values like the throughput of an app.
  • EventFieldListMetric collects a list of field values matching a certain event type. This helps generalize giant lists of fidelities, like purified fidelities over time.
  • DeliveryTimeMetric collects the time to deliver N purified pairs relative to reservation start. This helps generalized values like purified delivery of the entanglement purification protocol (RequestApp, etc.).

On Record Method

on_record() is an optional method that is called when the metric is recorded. It is only used by CounterPairMetric, where it updates the internal success and failure counters and updates the success rate.

Collect Context

CollectContext is a simple data class that lays out a template context to pass to the collect() method of each metric. It also tells the collect() method what it can expect when it gets called. It is only used by the collect_trial_metrics() method. This need some more work to be properly generalized or at least easily extended by the user.

Collect Trial Metrics Method

The collect_trial_metrics method is now more generalized for all types of metrics. All it does now is iterate through the list of metrics and calls the collect() method of each metric. It uses the created CollectContext object with the passed arguments, to pass those arguments through to the metrics collect() methods. However, these arguments are currently specific to certain protocols and are not currently generalized.

Tw1ZZLER added 30 commits June 18, 2026 15:25
I don't know if 2 Timelines will ever exist at once, but if so then this
might be a problem. For now it seems to be an okay solution
Comment thread sequence/utils/metrics/storage.py Fixed
@Tw1ZZLER Tw1ZZLER requested a review from caitaozhan July 1, 2026 17:43
Comment thread tests/utils/test_metrics.py Fixed
Tw1ZZLER and others added 2 commits July 1, 2026 13:36
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Comment thread sequence/utils/metrics/storage.py Outdated
Comment thread sequence/utils/metrics/__init__.py Outdated
Attributes:
owner_name: Node name for counter and fidelity metrics.
storage: In-memory store of recorded events for the trial.
delivery_owner: Node name for delivery-time metrics; defaults to ``owner_name``.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It currently defaults to None not owner_name

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also inline code entries are `` not ````

delivery_owner: str | None = None
target_pairs: int = 500
reservation_start_time: int | None = None
throughput: float | None = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove throughput

Comment thread sequence/utils/metrics/metric_types.py
Comment thread sequence/utils/metrics/__init__.py
Comment thread sequence/utils/metrics/__init__.py Outdated
Comment thread sequence/utils/metrics/__init__.py Outdated
Comment thread sequence/utils/metrics/builtins.py Outdated
Comment thread pyproject.toml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR continues the work from #368/#374 by introducing a centralized metrics tracking system under sequence.utils.metrics, with a registry of metric types/event types, an in-memory storage backend, and protocol instrumentation to record EG/EP/ES/delivery events during simulations.

Changes:

  • Adds a generalized metrics module (sequence/utils/metrics/*) with event type registration, metric type abstractions, and a metric registry.
  • Instruments entanglement generation, purification, and swapping protocols to call metrics.record(...) on success/failure.
  • Adds a comprehensive test suite covering recording, querying, collection, aggregation, and custom metric/event registration.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
tests/utils/test_metrics.py Adds unit tests validating metrics recording, collection, aggregation, and extensibility.
sequence/utils/metrics/init.py Implements public metrics API (enable/configure/record/collect/aggregate) and registers built-ins.
sequence/utils/metrics/event_types.py Introduces EventType value object and event type registry + built-in EventTypes.
sequence/utils/metrics/metric_types.py Adds metric type hierarchy (counters, rates, fidelity lists, delivery time) and CollectContext.
sequence/utils/metrics/registry.py Provides metric registration/lookup/reset helpers and counter lookup table.
sequence/utils/metrics/storage.py Adds in-memory record storage with basic query helpers.
sequence/utils/metrics/builtins.py Defines and registers built-in EG/EP/ES counters and fidelity/delivery/throughput metrics.
sequence/utils/init.py Exposes metrics from sequence.utils.
sequence/kernel/timeline.py Registers the Timeline instance as the metrics time provider.
sequence/entanglement_management/generation/generation_base.py Records EG failure events centrally.
sequence/entanglement_management/generation/single_heralded.py Records EG success events.
sequence/entanglement_management/generation/barret_kok.py Records EG success events.
sequence/entanglement_management/purification/bbpssw_circuit.py Records EP success/failure events.
sequence/entanglement_management/purification/bbpssw_bds.py Records EP success/failure events for BDS formalism.
sequence/entanglement_management/swapping/swapping_circuit.py Records ES success/failure events for circuit-based swapping.
sequence/entanglement_management/swapping/swapping_bds.py Records ES success/failure events for BDS swapping.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/metric_types.py Outdated
Comment thread sequence/utils/metrics/storage.py Outdated
Comment thread sequence/utils/metrics/__init__.py Outdated
Comment on lines +101 to +108
storage.append(
{
"event_type": event_type,
"owner_name": owner_name,
"sim_time": time_provider.now(),
**record_kwargs,
}
)
trial_values = trial[metric]
if list_metric_cap is not None:
trial_values = trial_values[:list_metric_cap]
all_values.extend(trial_values)
Tw1ZZLER and others added 8 commits July 1, 2026 15:20
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Centralized metrics tracking system

4 participants