Skip to content

Commit 4f2b767

Browse files
blasphemetheusclaude
blasphemetheus
and
claude
committed
feat: 8x the action_frame table from a live Mewtwo pair (9 -> 75 actions)
Recorded a real pair on the laptop — Mewtwo policy vs level-9 Fox CPU, 9189 frames — and merged its mapping. Coverage was the binding constraint on task #8's fix being useful for Mewtwo; it no longer is. Share of frames the table can normalize: fixture 9 actions 75 actions mewtwo_behind_response 9.7% 95.4% mewtwo_dtilt_uptilt_dense 8.4% 91.0% fox_multishine_closed 77.3% 85.9% gnw_neutral_dense 16.5% 82.6% gnw_movement_ledge 25.9% 74.4% mewtwo_approach_fair 12.1% 54.9% The pair validates cleanly: 100% agreement on action, on_ground and y over all 9189 frames, action_frame the ONLY field that shifts, every action a single consistent delta — reproducing on Mewtwo what phase 1 found on Fox. Cross-validation, the part that matters: the 6 actions measured by BOTH sources (24, 25, 29, 42, 323, 360) agree exactly, despite one being a frame-perfect Fox TAS loop and the other a Mewtwo bot in a real match. The convention is a property of Peppi-vs-libmelee, not of a character or a technique. Every delta ever measured is 0 or 1, now asserted. Trace emitter now SKIPS menu frames. The frame counter resets between games and idles in menus, so a trace spanning them carries duplicate frame numbers, and a duplicate silently pairs the wrong live frame to a parsed one. Cost when it happened: this very pair first scored 98.0% action agreement — close enough to look right, useless in fact — purely from post-game frames. Trimming to the in-game run gave exactly 100%. Emitting only IN_GAME/SUDDEN_DEATH makes that structural. Unknown menu_state still emits: a noisy trace is recoverable, a missing one is not. The anti-drift test becomes a CONSISTENCY check (the committed Fox pairs must never contradict the table) plus an exact pin against test/fixtures/statestream/action_frame_map.json, since the table is now broader than the vendored fixtures. Hand-editing @deltas without re-measuring still fails. Tests that used action 14 as "unmeasured" now use 252 (cliff catch) — 14 is measured (delta 1). 5 doctests + 6 properties + 470 tests, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 6c0d3da commit 4f2b767

5 files changed

Lines changed: 264 additions & 41 deletions

File tree

‎lib/exphil/data/action_frame_convention.ex‎

Lines changed: 111 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -16,40 +16,130 @@ defmodule ExPhil.Data.ActionFrameConvention do
1616
1717
## The table
1818
19-
Measured by `ExPhil.Eval.StateStreamDiff` from the committed pairs in
20-
`test/fixtures/statestream/` (task #8 phase 1). `delta` is
21-
`live_af - parsed_af`, constant per action and identical across both runs:
19+
Measured by `ExPhil.Eval.StateStreamDiff` from recorded pairs. `delta` is
20+
`live_af - parsed_af` and is a constant per action — every action ever
21+
measured has exactly one delta, and it is always 0 or 1.
2222
23-
live af == parsed af : 360, 365
24-
live af == parsed af + 1 : 24, 25, 29, 42, 323, 361, 366
23+
Two independent sources agree exactly on the 6 actions both cover
24+
(24, 25, 29, 42, 323, 360), despite one being a Fox TAS multishine loop and
25+
the other a Mewtwo policy fighting a level-9 CPU. So the convention is a
26+
property of Peppi-vs-libmelee, not of a character or technique.
2527
2628
There is no formula behind this — see `ExPhil.Eval.StateStreamDiff` for the
2729
two plausible rules that both fail.
2830
29-
## COVERAGE LIMIT — read before trusting this
30-
31-
The table covers **#{9} of #{399} action states**, all derived from two Fox
32-
multishine recordings. Unmeasured actions are passed through UNCHANGED,
33-
because the measured deltas are not extrapolable: they are mostly 1, but the
34-
two shine states (360, 365) are 0, and nothing observed so far predicts
35-
which an unmeasured action will be.
36-
37-
So this normalization strictly improves the 9 known actions and changes
38-
nothing else. It is NOT a complete fix for the state-stream shift. Widening
39-
it requires more recorded `.slp` + trace pairs, which requires Dolphin.
40-
Use `known?/1` and `unknown_actions/1` to see what a given workload would
41-
actually have normalized.
31+
## COVERAGE — good for Mewtwo/G&W play, still not total
32+
33+
The table covers **75 of 399** action states. Measured share of frames it
34+
normalizes, before (9 actions, Fox-only) and after adding the Mewtwo game:
35+
36+
| fixture | 9 actions | 75 actions |
37+
|---|---|---|
38+
| mewtwo_behind_response | 9.7% | 95.4% |
39+
| mewtwo_dtilt_uptilt_dense | 8.4% | 91.0% |
40+
| fox_multishine_closed | 77.3% | 85.9% |
41+
| gnw_neutral_dense | 16.5% | 82.6% |
42+
| gnw_movement_ledge | 25.9% | 74.4% |
43+
| mewtwo_approach_fair | 12.1% | 54.9% |
44+
45+
Unmeasured actions still pass through UNCHANGED, deliberately: the deltas
46+
are not extrapolable (a majority are 1, but many — including the shine
47+
states 360/365 and most hitstun/tumble states — are 0), so guessing would
48+
corrupt states that currently agree.
49+
50+
Widening further is cheap: record another pair (see
51+
`ExPhil.Eval.StateStreamTrace`) covering whatever the target character
52+
actually does, and merge. `unknown_actions/1` sizes the remaining gap for a
53+
specific workload.
4254
"""
4355

44-
# Measured, not assumed. Pinned against a fresh re-derivation from the
45-
# fixtures in test/exphil/data/action_frame_convention_test.exs, so this
46-
# table cannot silently drift away from the recordings it came from.
56+
# Measured, not assumed.
57+
#
58+
# Provenance — two independent sources, zero conflicts:
59+
# * 9 actions from the Fox multishine pairs in test/fixtures/statestream/
60+
# (a frame-perfect TAS loop)
61+
# * 72 from a Mewtwo-policy-vs-level-9-Fox-CPU game recorded 2026-07-26
62+
# (EXPHIL_STATE_TRACE=1, 9189 frames, 100% action/on_ground/y agreement)
63+
#
64+
# The 6 actions both measured (24, 25, 29, 42, 323, 360) agree exactly,
65+
# across different characters, inputs and situations — so the convention is
66+
# a property of Peppi-vs-libmelee, not of a character or a technique.
67+
#
68+
# Pinned by test/exphil/data/action_frame_convention_test.exs against
69+
# test/fixtures/statestream/action_frame_map.json, and cross-checked against
70+
# a live re-derivation from the committed pairs.
4771
@deltas %{
72+
12 => 1,
73+
14 => 1,
74+
15 => 0,
75+
16 => 0,
76+
18 => 0,
77+
20 => 0,
4878
24 => 1,
4979
25 => 1,
80+
26 => 1,
81+
27 => 1,
82+
28 => 0,
5083
29 => 1,
84+
35 => 1,
85+
39 => 0,
86+
40 => 1,
87+
41 => 1,
5188
42 => 1,
89+
43 => 1,
90+
44 => 0,
91+
47 => 0,
92+
48 => 1,
93+
49 => 1,
94+
53 => 0,
95+
56 => 0,
96+
57 => 0,
97+
60 => 0,
98+
63 => 0,
99+
64 => 0,
100+
65 => 0,
101+
66 => 0,
102+
67 => 0,
103+
68 => 0,
104+
69 => 0,
105+
70 => 0,
106+
71 => 0,
107+
72 => 0,
108+
74 => 0,
109+
76 => 0,
110+
78 => 0,
111+
79 => 0,
112+
80 => 0,
113+
84 => 0,
114+
85 => 0,
115+
86 => 0,
116+
88 => 0,
117+
180 => 0,
118+
181 => 0,
119+
212 => 1,
120+
213 => 0,
121+
214 => 1,
122+
216 => 0,
123+
217 => 0,
124+
218 => 0,
125+
219 => 0,
126+
220 => 0,
127+
221 => 0,
128+
222 => 0,
129+
233 => 0,
130+
234 => 0,
131+
235 => 0,
132+
236 => 0,
133+
264 => 1,
52134
323 => 1,
135+
341 => 0,
136+
342 => 1,
137+
345 => 1,
138+
346 => 0,
139+
347 => 1,
140+
350 => 1,
141+
351 => 0,
142+
359 => 0,
53143
360 => 0,
54144
361 => 1,
55145
365 => 0,

‎lib/exphil/eval/state_stream_trace.ex‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,12 +106,29 @@ defmodule ExPhil.Eval.StateStreamTrace do
106106
def maybe_emit(state)
107107

108108
def maybe_emit(%GameState{} = state) do
109-
if enabled?(), do: emit(state, port())
109+
if enabled?() and in_game?(state), do: emit(state, port())
110110
state
111111
end
112112

113113
def maybe_emit(other), do: other
114114

115+
# Menu frames must NOT be traced. The frame counter resets between games and
116+
# idles during menus, so a trace spanning menus contains DUPLICATE frame
117+
# numbers — and a duplicate silently pairs the wrong live frame to a parsed
118+
# one. Observed cost: a Mewtwo pair scored 98.0% action agreement (looks
119+
# nearly right, is unusable) purely from post-game frames after the counter
120+
# reset; trimming to the in-game run took it to exactly 100%.
121+
#
122+
# libmelee Menu values: IN_GAME = 2, SUDDEN_DEATH = 3.
123+
@in_game_menu_states [2, 3]
124+
125+
defp in_game?(%GameState{menu_state: menu}) when is_integer(menu),
126+
do: menu in @in_game_menu_states
127+
128+
# Unknown/absent menu_state: emit rather than silently record nothing —
129+
# a noisy trace is recoverable, a missing one is not.
130+
defp in_game?(_state), do: true
131+
115132
defp emit(%GameState{frame: frame, players: players}, port)
116133
when is_integer(frame) and is_map(players) do
117134
case players[port] do

‎test/exphil/data/action_frame_convention_test.exs‎

Lines changed: 48 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,13 @@ defmodule ExPhil.Data.ActionFrameConventionTest do
99
@dir "test/fixtures/statestream"
1010

1111
describe "the table is the measurement" do
12-
test "static deltas match a fresh derivation from the committed pairs" do
13-
# THE anti-drift test. AFC hardcodes the table so it is usable at
14-
# runtime without parsing fixtures; this proves the hardcoded copy still
15-
# equals what the recordings actually say. If someone edits @deltas by
16-
# hand, or a Peppi/libmelee upgrade moves the convention, this fails
17-
# here rather than silently mis-normalizing every policy.
12+
test "committed pairs never CONTRADICT the table" do
13+
# THE anti-drift test. The table is broader than the committed pairs
14+
# (most of it comes from a Mewtwo recording too large to vendor), so
15+
# this is a consistency check rather than equality: every action these
16+
# fixtures measure must match what the table claims. A Peppi/libmelee
17+
# upgrade that moves the convention fails here instead of silently
18+
# mis-normalizing every policy.
1819
derived =
1920
for name <- ["fox_ms_frame1", "fox_ms_float"], reduce: %{} do
2021
acc ->
@@ -24,12 +25,40 @@ defmodule ExPhil.Data.ActionFrameConventionTest do
2425
Map.merge(acc, Map.new(report.mapping, fn {action, m} -> {action, m.delta} end))
2526
end
2627

27-
assert derived == AFC.deltas()
28+
refute Enum.empty?(derived)
29+
30+
for {action, delta} <- derived do
31+
assert AFC.deltas()[action] == delta,
32+
"fixture says action #{action} has delta #{delta}, table says " <>
33+
"#{inspect(AFC.deltas()[action])}"
34+
end
35+
end
36+
37+
test "table matches the pinned mapping fixture exactly" do
38+
# action_frame_map.json is the recorded measurement, regenerable with
39+
# scripts/diff_state_streams.exs --json. Hand-editing @deltas without
40+
# re-measuring fails here.
41+
pinned =
42+
"#{@dir}/action_frame_map.json"
43+
|> File.read!()
44+
|> Jason.decode!()
45+
|> Map.fetch!("deltas")
46+
|> Map.new(fn {k, v} -> {String.to_integer(k), v} end)
47+
48+
assert pinned == AFC.deltas()
49+
end
50+
51+
test "covers the 75 measured actions" do
52+
assert AFC.coverage() == 75
53+
54+
# The Fox-derived originals must survive every later merge.
55+
for action <- [24, 25, 29, 42, 323, 360, 361, 365, 366] do
56+
assert AFC.known?(action)
57+
end
2858
end
2959

30-
test "covers exactly the 9 measured actions" do
31-
assert AFC.coverage() == 9
32-
assert Map.keys(AFC.deltas()) |> Enum.sort() == [24, 25, 29, 42, 323, 360, 361, 365, 366]
60+
test "deltas are only ever 0 or 1" do
61+
assert AFC.deltas() |> Map.values() |> Enum.uniq() |> Enum.sort() == [0, 1]
3362
end
3463
end
3564

@@ -55,11 +84,13 @@ defmodule ExPhil.Data.ActionFrameConventionTest do
5584

5685
describe "coverage limits are explicit, not silent" do
5786
test "unmeasured actions pass through unchanged" do
58-
# The deltas are NOT extrapolable (mostly 1, but 360/365 are 0), so an
59-
# unmeasured action must not be guessed at.
60-
assert AFC.live_to_parsed(14, 5) == 5
61-
assert AFC.live_to_parsed(252, 0) == 0
62-
refute AFC.known?(14)
87+
# The deltas are NOT extrapolable (a majority are 1, but many — the
88+
# shine states 360/365, most hitstun/tumble — are 0), so an unmeasured
89+
# action must not be guessed at. 252/253 are cliff catch/wait, which no
90+
# recorded pair has covered yet.
91+
assert AFC.live_to_parsed(252, 5) == 5
92+
assert AFC.live_to_parsed(253, 0) == 0
93+
refute AFC.known?(252)
6394
end
6495

6596
test "negative action_frame sentinels are never adjusted" do
@@ -73,8 +104,8 @@ defmodule ExPhil.Data.ActionFrameConventionTest do
73104
end
74105

75106
test "unknown_actions/1 sizes the gap for a workload" do
76-
assert AFC.unknown_actions([24, 14, 365, 252, 14]) == [14, 252]
77-
assert AFC.unknown_actions([24, 365]) == []
107+
assert AFC.unknown_actions([24, 253, 365, 252, 253]) == [252, 253]
108+
assert AFC.unknown_actions([24, 365, 14]) == []
78109
end
79110

80111
test "known?/1 is honest about what was measured" do

‎test/exphil/embeddings/af_convention_test.exs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,9 @@ defmodule ExPhil.Embeddings.AfConventionTest do
7676
end
7777

7878
test "unmeasured actions are untouched under :live" do
79-
live = PlayerEmbed.embed_frame_info(player(14, 5), live_config())
80-
parsed = PlayerEmbed.embed_frame_info(player(14, 5), parsed_config())
79+
# 252 = cliff catch, which no recorded pair covers yet.
80+
live = PlayerEmbed.embed_frame_info(player(252, 5), live_config())
81+
parsed = PlayerEmbed.embed_frame_info(player(252, 5), parsed_config())
8182

8283
assert Nx.to_flat_list(live) == Nx.to_flat_list(parsed)
8384
end

‎test/fixtures/statestream/action_frame_map.json‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
{
2+
"_comment": "parsed<->live action_frame convention (task #8, GOTCHAS #81). delta = live_af - parsed_af. Pins ExPhil.Data.ActionFrameConvention. Regenerate with scripts/diff_state_streams.exs --json on a 100%-agreement pair.",
3+
"_sources": [
4+
"test/fixtures/statestream/fox_ms_frame1 + fox_ms_float (Fox multishine, 9 actions)",
5+
"Mewtwo policy vs level-9 Fox CPU, 2026-07-26, 9189 frames, 100% action/on_ground/y agreement (72 actions)"
6+
],
7+
"deltas": {
8+
"12": 1,
9+
"14": 1,
10+
"15": 0,
11+
"16": 0,
12+
"18": 0,
13+
"20": 0,
14+
"24": 1,
15+
"25": 1,
16+
"26": 1,
17+
"27": 1,
18+
"28": 0,
19+
"29": 1,
20+
"35": 1,
21+
"39": 0,
22+
"40": 1,
23+
"41": 1,
24+
"42": 1,
25+
"43": 1,
26+
"44": 0,
27+
"47": 0,
28+
"48": 1,
29+
"49": 1,
30+
"53": 0,
31+
"56": 0,
32+
"57": 0,
33+
"60": 0,
34+
"63": 0,
35+
"64": 0,
36+
"65": 0,
37+
"66": 0,
38+
"67": 0,
39+
"68": 0,
40+
"69": 0,
41+
"70": 0,
42+
"71": 0,
43+
"72": 0,
44+
"74": 0,
45+
"76": 0,
46+
"78": 0,
47+
"79": 0,
48+
"80": 0,
49+
"84": 0,
50+
"85": 0,
51+
"86": 0,
52+
"88": 0,
53+
"180": 0,
54+
"181": 0,
55+
"212": 1,
56+
"213": 0,
57+
"214": 1,
58+
"216": 0,
59+
"217": 0,
60+
"218": 0,
61+
"219": 0,
62+
"220": 0,
63+
"221": 0,
64+
"222": 0,
65+
"233": 0,
66+
"234": 0,
67+
"235": 0,
68+
"236": 0,
69+
"264": 1,
70+
"323": 1,
71+
"341": 0,
72+
"342": 1,
73+
"345": 1,
74+
"346": 0,
75+
"347": 1,
76+
"350": 1,
77+
"351": 0,
78+
"359": 0,
79+
"360": 0,
80+
"361": 1,
81+
"365": 0,
82+
"366": 1
83+
}
84+
}

0 commit comments

Comments
 (0)