Skip to content

TESTING.md

Latest commit

 

History

History
185 lines (157 loc) · 10.4 KB

TESTING.md

File metadata and controls

185 lines (157 loc) · 10.4 KB
Raw

Testing strategy

"A carpenter, on arriving at a site, builds the workstation for the day's project from the materials and tools at hand. We should be the same: while implementing rubyrs, build the tools that will guarantee its quality in the future."

This document explains how we keep rubyrs honest as it grows, and the pipeline we are building to scale that.

Four layers

┌─────────────────────────────────────────────────────────────────┐
│  Layer 1: unit tests (in modules, #[cfg(test)])                  │
│  - Tight Rust-level checks for individual functions              │
│  - Currently sparse; we lean on layers 2/3/4                     │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│  Layer 2: integration fixtures (tests/integration.rs)            │
│  - .rb file + .expected golden file for stdout                   │
│  - tests/fixtures/errors/ for .expected_err (stderr) cases       │
│  - `cargo test` execs the rubyrs binary, diffs                   │
│  - UPDATE_EXPECTED=1 cargo test regenerates the goldens          │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│  Layer 3: public API smoke tests (tests/embed.rs)                │
│  - Calls Runtime/Config/register_fn/set_stdout/format_trap       │
│  - Pins the embedding API surface so accidental breakage shows   │
│    up in CI; also exercises resource caps (fuel/heap/frames)     │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│  Layer 4: ruby/spec coverage (planned, see below)                │
│  - Spec files from upstream ruby/spec, mechanically translated   │
│  - Run on rubyrs AND CRuby; compare PASS counts                  │
│  - SPEC_STATUS.md is auto-generated; coverage drives credibility │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│  Layer 5: cargo-fuzz soak (nightly)                              │
│  - libfuzzer-sys harness over Runtime::eval + parser path        │
│  - Discovers panics / ICEs no hand-written fixture would find    │
│  - Findings get pinned as Layer 2 / 3 fixtures and fixed         │
│  - See docs/FUZZING.md                                           │
└─────────────────────────────────────────────────────────────────┘

Layers 2 and 3 are hand-curated — we choose the cases. Layer 4 is the antidote: an external standard chooses the cases for us. Layer 5 closes a different loop: nothing chooses the cases — the fuzzer mutates bytes until the binary crashes.

Why ruby/spec, not our own tests

ruby/spec is the de facto executable standard for Ruby semantics. CRuby, JRuby, and TruffleRuby all run it. Picking it as our oracle means:

  • No bias — we don't get to handpick what we test.
  • Credibility — "we pass X% of ruby/spec" is the only meaningful compatibility metric outside the CRuby team's blessing.
  • Forced discipline — we can't skip the hard semantic edges.

The ingestion pipeline

ruby/spec uses mspec, a custom test framework. Running mspec inside rubyrs requires roughly all of Ruby (classes, modules, Symbol, interpolation, exception hierarchy, method_missing, file I/O, ...) — that's the TruffleRuby/JRuby long-tail path, not a near-term option.

Instead, we treat ruby/spec as a corpus and ingest it mechanically:

ruby/spec  (git submodule — pending)
   │
   ▼  ❶ crates/rubyrs-spec-extract  (Rust crate; ruby_prism walk)
   │   Reads an upstream spec file, rewrites the recognised
   │   matcher patterns (v0.1: `expr.should == val`), passes
   │   the rest through verbatim so a human can finish.
   │   See crates/rubyrs-spec-extract/README.md for the
   │   pattern surface.
   │
   ▼  crates/rubyrs/spec/ruby/<feature>_spec.rb
   │   The extracted file lands alongside the existing
   │   hand-translated set and is picked up by the runner.
   │
   ▼  ❷ tests/ruby_spec.rs  (existing micro-runner)
   │   Runs every spec file end-to-end through `Runtime`,
   │   counting pass/fail per example. Today the file must
   │   pass cleanly; a tagged-divergent lane is future work.
   │
   ▼  ❸ docs/SPEC_STATUS.md (auto-generated)
       Regenerated by `UPDATE_SPEC_STATUS=1 cargo test -p rubyrs
       --test spec_status`; the same test fails CI when the
       on-disk file drifts from the corpus. One section per
       class with file count, example count, and skipped-trace
       breakdown by `polish.py` category.

Each piece is a small, focused tool. The pipeline grows by teaching spec_extract new patterns, not by re-doing manual work.

Pipeline versions

rubyrs-spec-extract will support more mspec patterns over time. We treat each pattern as a version increment:

Version Recognises Estimated reach Status
v0.1 expr.should == val, require_relative strip ~10–20% of spec files shipped
v0.2 should_not == val, predicate matchers (.should.foo? / .should_not.foo?), -> { ... }.should.raise(X) lambda lowering (incl. M::Cls class paths) +20–25% shipped
v0.3 before :each body lift into sibling its, mock_int(LITERAL_INT) substitution, skip-log header listing unrewritten patterns per file +10% shipped
v0.4 it_behaves_like :NAME, args... cross-file inlining (via --shared <path> CLI flag); @method / @method2 / ... substitution; recognisers run on the substituted body +20% shipped
... ... approaching mspec full ...

Anything the extractor doesn't recognise — at any version — passes through unchanged; the human reviewer sees what's still hand-translation territory in the diff against upstream. A "skipped and logged" report (per the original ingestion plan) is deferred to v0.3 alongside the before / after hook handling, where the cost-to-write-the-log starts paying off (predicate-heavy v0.1 / v0.2 files extract mostly cleanly, so a per-line skip log adds little signal there).

Current state

Live counts per class, file, and skip category live in SPEC_STATUS.md — auto-generated by cargo test -p rubyrs --test spec_status (regenerate with UPDATE_SPEC_STATUS=1 cargo test -p rubyrs --test spec_status). The translation conventions live in crates/rubyrs/spec/README.md and the mechanical-extraction pipeline in crates/rubyrs-spec-extract/README.md.

Every example must pass — there is no "tagged divergent" lane yet. Skipped upstream it blocks are noted in the spec file's top-of-file comment with the reason (out of subset / out of master / fixtures-dependent). When master lands a feature that unblocks a previously-skipped block, the comment becomes the ratchet: un-skip and re-test.

Progress goes by extractor-then-curate: run crates/rubyrs-spec-extract (v0.4 shipped) on the upstream spec

  • any shared bodies, then pipe through crates/rubyrs-spec-extract/scripts/polish.py to drop fixture / mock / unimplemented-method blocks. Hand-translated files remain as a validation oracle — "does the tool produce the same output as a human did?" — and are kept in lockstep when the extractor gains new patterns.

Workflow for adding a feature

The intended pull request flow:

  1. Pick a feature from ROADMAP.md, or a spec directory that is partly red.
  2. Implement the language feature.
  3. Write at least one hand-crafted fixture in tests/fixtures/ to lock in observable behaviour. This stays small and human-readable.
  4. If the feature unlocks a previously-skipped block, regenerate the affected spec via the extractor + polish pipeline (see crates/rubyrs-spec-extract/README.md) and drop the skip from crates/rubyrs/spec/ruby/<feature>_spec.rb.
  5. Run cargo test. New tests should pass; old ones must still pass.
  6. Regenerate SPEC_STATUS.md: UPDATE_SPEC_STATUS=1 cargo test -p rubyrs --test spec_status. CI's spec_status_is_up_to_date test fails otherwise.
  7. Commit. Reference the spec coverage delta in the commit body.

This loop means every new feature is automatically graded against ruby/spec, without anyone manually picking what to test.

Why we are not running mspec inside rubyrs (yet)

Goal of mspec-inside-rubyrs is on the long-term ROADMAP.md under "Run mspec inside rubyrs". It only becomes meaningful once we have ~95% of mspec's own dependencies running — Symbol, interpolation, Module, exception classes, method_missing, IO, Comparable, Enumerable as a mixin. We get there by following the SPEC_STATUS.md report; once it crosses a threshold the switch is mechanical.

Until then, the extractor approach gives us:

  • Faster iteration (no Ruby host bootstrap)
  • Identical CRuby/rubyrs comparison, byte-level
  • A real number to report, every commit