Skip to content

Expand reasoning-failure examples and negative fixtures across ten families #106

Description

@mdheller

Context

Follow-up from #38 after #45 landed the first Reasoning Failure ontology tranche.

#45 added:

  • Platform/ReasoningFailure/reasoning-failure.ttl
  • shapes/reasoning_failure.shacl.ttl
  • contexts/reasoning-failure.context.jsonld
  • examples/reasoning_failure_demo.ttl
  • catalog/reasoning_failure_registry.ttl
  • docs/reasoning-failure-ontology-v0.1.md

The core semantic surface exists, but #38 also required positive and negative examples across at least ten failure families.

Scope

Add broader fixtures for reasoning failure coverage:

  • positive examples across at least ten failure families
  • negative fixtures for fail-closed behavior
  • explicit reversal / inverse relation fixture
  • explicit two-hop shortcut exploitation fixture
  • missing invariant fixture
  • missing verifier fixture
  • missing evidence receipt fixture
  • missing residual risk fixture
  • ambiguous relation direction fixture

Candidate failure families

  • cognitive state / stale context
  • bias and framing
  • formal relation reasoning
  • exact string / tokenization / arithmetic
  • impossible or contradictory task handling
  • social reasoning and anthropomorphic trust
  • code perturbation robustness
  • causal or temporal reasoning
  • multimodal / embodied / spatial reasoning
  • scientific reasoning
  • multi-agent misalignment / termination / obfuscation

Acceptance criteria

  • Fixtures are placed so invalid examples do not break the repo-wide examples/ SHACL gate.
  • A focused validator proves valid examples pass and invalid fixtures fail.
  • make validate or a named make target includes the focused validator.
  • make all remains green.
  • Documentation states which failure families are covered.

Non-goals

  • No runtime perturbation runner.
  • No model evaluation service.
  • No ledger storage.
  • No UI or search implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions