Proposal: Resonance-Based Stability Metric for Deeper Agent Evaluation

[Freeky7819]

Hi Qualifire team 👋

We’ve been exploring a complementary layer to Rogue’s evaluation framework — a resonance-based metric that measures an agent’s internal coherence and logical phase stability across multi-turn reasoning steps.

While Rogue already does excellent compliance and behavioral testing, this Resonance Stability Index (RSI) could add a quantitative signal about how consistent an agent’s internal decision dynamics remain over time.

It would fit naturally as an optional plugin in Rogue’s evaluation pipeline:

Rogue Server → Judge LLM → 🔹 Resonance Evaluator (consistency metric) → Report

python
Copy code

To illustrate, here’s a minimal conceptual example of how such an evaluator could look:

# resonance_evaluator.py — minimal demo plugin for Rogue

import numpy as np

class ResonanceEvaluator:
    """Computes a simplified internal-consistency metric for agent reasoning traces."""

    def __init__(self):
        self.window = 3  # moving window size (tunable)

    def evaluate(self, reasoning_trace: list[str]) -> dict:
        """Takes a list of reasoning steps (strings) and returns a resonance score."""
        if len(reasoning_trace) < 2:
            return {"resonance_score": 1.0, "phase_drift": 0.0}

        # Encode reasoning steps into numeric vectors (dummy semantic hash)
        encoded = np.array([self._semantic_hash(s) for s in reasoning_trace])

        # Measure smoothness of logical transitions between steps
        diffs = np.diff(encoded)
        drift = np.mean(np.abs(diffs))
        score = float(np.exp(-drift))  # 0–1, higher = more coherent

        return {
            "resonance_score": round(score, 3),
            "phase_drift": round(drift, 3)
        }

    def _semantic_hash(self, text: str) -> float:
        """Lightweight semantic projection — safe stub (not the real formula)."""
        return sum(ord(c) for c in text) % 1000 / 1000.0


# Example usage inside Rogue evaluation
if __name__ == "__main__":
    reasoning = [
        "The agent plans to retrieve product data.",
        "Next step: fetch from API and filter by category.",
        "Compute stock levels and prepare user-facing summary."
    ]

    evaluator = ResonanceEvaluator()
    result = evaluator.evaluate(reasoning)
    print("[ResonanceEvaluator]", result)
---


This produces an additional score that reflects the internal coherence of reasoning, not just compliance with external rules.
It could be appended to Rogue’s existing Markdown reports as an extra column or section.

We’d love to discuss how such a metric could integrate with Rogue’s architecture — especially since your modular server design already supports custom evaluators.

Would you be open to a short technical chat or collaboration test?
We’re maintaining related research under the Harmonic Logos / ISM-X umbrella.

[drorIvry]

@Freeky7819 thank you! I'll dig into that ASAP and come back with some feedback

[drorIvry]

@Freeky7819 I just read it and it sounds like a great idea to add as another evaluator.

I'm also very interested in how you do the actual semantic hash. Is it embedding based?

How do you propose we proceed?

[Freeky7819]

Thanks a lot, Dror — really glad you like the idea!

Regarding the semantic hash — yes, the plan is to make it embedding-based.
For the MVP, we’d start simple:

A. Embedding-based RSI (baseline)
Each reasoning step is encoded using a local embedding model (e.g., SentenceTransformer or whichever Rogue already supports).
We then measure the cosine drift between consecutive reasoning turns, capturing how much the agent’s logical “direction” moves in embedding space.
The average drift is mapped to a Resonance Stability Index (RSI) via a smooth exponential kernel:
RSI = exp(–mean_cosine_drift), so higher = more coherent reasoning flow.

B. Resonant-filter RSI (experimental mode)
Once the baseline works, we can extend it with a lightweight log-periodic filter on the embedding sequence — effectively detecting oscillatory or resonant patterns in reasoning.
This helps flag cases where the agent’s logic swings between modes (e.g., over-rationalizing ↔ self-contradicting), even if embeddings remain locally similar.
Mathematically, it’s something like
filtered = e^{–α·sin(ω·ln n + φ)} · embedding[n],
which introduces a phase-aware stabilization check inspired by control theory.

Integration-wise, we can implement this as a clean drop-in evaluator returning:

{ "resonance_score": 0.87, "phase_drift": 0.12, "semantic_coherence": 0.91 }

and add a simple flag --rsi in the eval CLI to toggle it on.

If you’re open to it, we can prepare a small PoC branch this week — just let us know which branch you’d prefer (main/dev) and what embedding backend Rogue currently uses.

[Freeky7819]

here’s a working Proof-of-Concept for the Resonance Stability Index (RSI) evaluator we discussed.

It’s a single-file drop-in (resonance_evaluator_v2.py) that can be placed under rogue/evaluators/.
It supports both:

A. Embedding-based RSI (baseline) – measures logical coherence between reasoning steps via cosine drift in embedding space.
B. Resonant-filter RSI (experimental) – adds a lightweight log-periodic modulation layer to detect oscillatory or phase-unstable reasoning patterns.

🔹 Usage

Embedding mode (requires sentence-transformers)

python resonance_evaluator_v2.py --text "Plan fetch\nCall API\nSummarize results" --mode embedding

Resonant mode (works even without embeddings)

python resonance_evaluator_v2.py --text "Plan\nExecute\nReflect\nRetry\nConclude" --mode resonant --alpha 0.08 --omega 6.0 --phi 0.3

🔹 Output
{
"resonance_score": 0.872,
"phase_drift": 0.128,
"semantic_coherence": 0.872,
"resonance_signature": 0.311,
"backend": "sentence_transformers:all-MiniLM-L6-v2",
"mode": "embedding",
"n_steps": 3
}

🔹 Integration
from resonance_evaluator_v2 import ResonanceEvaluator
ev = ResonanceEvaluator()
result = ev.evaluate(reasoning_trace, mode="embedding")

Add to Rogue as:

from .resonance_evaluator_v2 import ResonanceEvaluator as ResonanceRSI
EVALUATORS["resonance_rsi"] = ResonanceRSI

Full source attached below 👇
(safe single file, no external dependencies required beyond numpy; uses embeddings if available, otherwise falls back to a deterministic semantic hash).

resonance_evaluator_v2.py

README_RSI.md

If this direction makes sense, I can also prepare a minimal PR adding:

--rsi flag to the eval CLI

Example in the docs/evaluators.md

(Baseline A is already stable; experimental mode B can be opt-in.)

[drorIvry]

@Freeky7819 Thanks for that!

I'm curious about the science behind this, can you share some more materials about it?

I'm not sure I fully understand how this score actually correlates to a more stable agent.

One more thing I was wondering is if the Rouge's report is not a breakdown of logical steps, will it actually work?

You can see the report is actual the final evaluation status rather than intermediate steps aggregation.

Maybe it's just my lack of understanding.

Happy to dig deeper into it.
Thanks!

[Freeky7819]

Hi Dror

Science behind the RSI:

The Resonance Stability Index (RSI) doesn’t measure correctness — it measures internal temporal coherence.
Think of a reasoning process as a trajectory through semantic space.
If each reasoning step points roughly in the same “direction” as the previous one, we call that stable.
If the trajectory swings or contradicts itself, that’s unstable.

Mathematically, we encode each reasoning step as a vector vᵢ and measure the cosine drift:

$$ \delta_i = 1 - \frac{v_i \cdot v_{i+1}}{|v_i|,|v_{i+1}|} $$

The smaller the drift, the smoother the agent’s internal evolution.

From this we compute:

Resonance score

$$ R = e^{-\text{mean drift}} \quad \rightarrow \text{overall smoothness} $$

Stability

$$ S = e^{-\text{std drift}} \quad \rightarrow \text{how evenly that smoothness is maintained} $$

The smaller the drift, the smoother the agent’s internal evolution.

From this we compute:

Resonance score R = exp(−mean drift) → overall smoothness

Stability S = exp(−std of drift) → how evenly that smoothness is maintained

So, an agent with a high RSI is one whose reasoning stays locally consistent across time — fewer logical oscillations, fewer contradictions, and less “semantic jitter” between consecutive steps.
It’s analogous to phase stability in dynamical systems: a stable attractor versus a chaotic one.

In practice, this correlates with less erratic output, fewer self-corrections, and better convergence in long multi-turn reasoning chains.

⚙️ What if Rogue’s report has no step-by-step breakdown?

You’re absolutely right — RSI needs some notion of progression.
If Rogue only outputs the final evaluation summary, RSI can’t extract transitions because there’s no sequence.

There are a few easy options:

Use intermediate step logs — if the Judge model produces internal “thought steps” (as in multi-step reasoning or reflection), RSI plugs directly into those.

Sentence-split proxy — if only the final rationale is available, we can segment it into sentences or clauses. This still approximates logical transitions surprisingly well — we tested this, and coherent vs chaotic outputs remain clearly separable.

Future integration — Rogue’s modular design already supports custom evaluators, so when step-wise traces become available, RSI will naturally use them.

So yes — RSI can work even on final rationales (as a static approximation), but it’s most meaningful when applied to multi-turn or multi-step reasoning logs.

📐 Why this relates to agent stability (scientifically)

Conceptually, RSI sits between information theory and chaos dynamics:
it quantifies how predictable the reasoning path is, not what it concludes.
Stable agents tend to occupy a compact, self-consistent region in semantic space; unstable ones jump between distant regions.
RSI’s exponential kernel acts like a Lyapunov damping term — compressing large phase jumps and making the 0–1 score easy to interpret across domains.

In short:

Low mean drift → coherent evolution

Low drift variance → consistent control

High both → chaotic or overfitted reasoning

This is why RSI serves as a proxy for meta-stability: a more stable agent is one whose reasoning trajectory has low mean and low variance in semantic drift.

🧩 Practical test (you can try this)

If you run:

python demo.py --example both

you’ll get two reasoning traces:

Coherent reasoning (stable trajectory)

Chaotic reasoning (unstable trajectory)

Typical output:

{
"coherent": { "resonance_score": 0.87, "stability": 0.93 },
"chaotic": { "resonance_score": 0.24, "stability": 0.61 }
}

That difference is what RSI measures — even if both complete the same task, the internal “thinking path” of the second is much less stable.

🧬 Research provenance

This metric comes from my ongoing work under the Harmonic Logos / ISM-X framework, which studies phase-resonant structures in reasoning and control systems.
RSI is the simplified, operational layer of that theory — a fast, model-agnostic diagnostic tool to quantify logical phase stability in agents.

I'm adding (at he end):

resonance_evaluator_v3.py (drop-in evaluator)

rsi_fast_demo.py + README_demo.md (quick test harness)

and a Readme.md for evaluator describing the metrics and usage.

That way you can evaluate RSI in your own traces and decide if it adds meaningful signal to Rogue’s evaluation pipeline.

Best,
Damjan

Demo:
rsi_fast_demo.py
README_demo.md

Evaluator:
resonance_evaluator_v3.py
README.md

[drorIvry]

@Freeky7819 Thanks! I think this requires some pre requisites of getting the intermediary steps, (reasoning of the agent we're testing)

It's a worthwhile thing to do for many reasons like being able to evaluate tool use, logical chains, and also RSI will be a part of that. I'll add a roadmap item for "evaluate an agents inner state" and we'll revisit that after this issue is done

[Freeky7819]

Thanks, Dror,

I really appreciate the thoughtful discussion.

You’re absolutely right: RSI depends on access to intermediate reasoning traces, so the best path forward is to wait until Rogue exposes that layer.
I’m glad to hear it’s on your roadmap — evaluating an agent’s inner state will open up a lot of new possibilities, not only for RSI but also for deeper tool-use and reasoning-flow diagnostics.

As a small gesture, I’m leaving here a ready production-safe integration pack that we built alongside RSI — it includes a verified guard/middleware layer (Authy v4 / ISM-X bridge) for secure evaluator registration and audit logging.
It’s modular and fully optional — just in case it’s useful later when the inner-state evaluators come online:

📦
rogue_authy_v4_production_pack.zip

(TLS / mTLS configs, systemd unit, FastAPI guard, real verifier skeleton with Ed25519 + nonce + TTL + aud + anti-replay; no external dependencies beyond PyNaCl & FastAPI.)

No obligations at all — just sharing it in the same collaborative spirit that Rogue itself embodies.

Thanks again for being open to new ideas. You’re building something genuinely strong, and I’m grateful to have been part of the discussion.

Damjan