[Dev] ci: guard examples/multimodal_dev with a unit-test bucket

[BestJuly]

• I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Runs the examples/multimodal_dev/tests/ suite as its own H100 unit-test bucket, and repairs the two things that stopped that suite from being a usable guard.

examples/multimodal_dev tracks megatron.core APIs closely — rope_utils, GPTModel, TransformerConfig, the GPT layer specs — but it lives outside tests/unit_tests, so nothing in CI exercises it. Core changes have therefore landed without anyone noticing they broke the example.

That is not hypothetical: it is broken on dev today. The fused-mRoPE change (#5962) made apply_rotary_pos_emb read config.mrope_section and config.apply_rope_fusion directly, and test_mrope_parity.py::test_vision_rope_wrapper_forwards_max_seqlen_to_thd drives that code with a SimpleNamespace stub written before those fields existed:

AttributeError: 'types.SimpleNamespace' object has no attribute 'mrope_section'
megatron/core/models/common/embeddings/rope_utils.py:93

Measured on dev @ 9304a2564, 1×8×H100: 1 failed, 45 passed (identical at 1 rank and at 8).

Three commits:

1. Repair the stale config stub. The stub now carries mrope_section=None, apply_rope_fusion=False, pinning the test to the unfused legacy THD path it was written for. No change to megatron/core.

2. Make the standalone harnesses real tests. Four of the eight files (test_cp_correctness.py, test_cp_thd_correctness.py, test_thd_correctness.py, test_vision_patch_merger_parity.py) were argparse + main() scripts with no test_* functions — pytest imported them and collected nothing, so the BSHD-vs-THD, CP-vs-CP=1 and patch-merger parity checks they implement only ever ran by hand. Each comparison body is factored into a reusable function that both a pytest entry point and the original main() call, so the CLI and the tests cannot drift; a per-module DEFAULTS dict backs both the argparse defaults and the test parameters. CP sizes that do not divide the world size skip rather than fail, and every module leaves the model-parallel groups torn down, so the files work at any --nproc-per-node and share one pytest process. The __main__ entry points are unchanged in behaviour and were re-verified after the refactor. Net: +7 executing tests.

3. Wire up the bucket. New test_case: [examples/multimodal_dev/tests/**/*.py] in tests/test_utils/recipes/h100/unit-tests.yaml, plus a local conftest.py. The bucket is collected outside tests/unit_tests/, so tests/unit_tests/conftest.py does not apply to it; the local conftest supplies exactly what tests/unit_tests/run_ci_test.sh depends on — the --experimental option its second pytest pass passes, and the pytest_sessionfinish hook that maps pytest's "no tests collected" (exit 5) to success.

Scope of the bucket

environment: [dev], tag: [latest], deliberately:

• tag: latest only — the legacy ref pins an older mcore and an older copy of this example, so the pairing this bucket guards does not exist there.
• environment: dev only — validated against the dev container; lts is unverified and an unverified entry risks a red bucket on older TE. Widening is a one-line change once it has been observed green.

gb200 is not wired up: selection there is marker-driven (launch_on_gb200) and no file in this suite carries the marker, so the bucket would launch an empty job.

Validation (1×8 H100)

	Result
dev @ 9304a2564, baseline	1 failed, 45 passed
This branch, 8 ranks	53 passed

The exact CI entry point was then run with the tree bind-mounted at /opt/megatron-lm so the script's hardcoded paths resolve as they do in CI:

bash tests/unit_tests/run_ci_test.sh --tag latest --environment dev \
  --bucket "examples/multimodal_dev/tests/**/*.py" --log-dir /tmp/ci-logs

53 passed, 64 warnings in 40.11s                  # pass 1: -m 'not experimental and not flaky_in_dev'
collected 53 items / 53 deselected / 0 selected   # pass 2: --experimental
RUN_CI_TEST_EXIT=0

Pass 2 is what the new conftest exists for: --experimental is accepted and the resulting exit code 5 is mapped to 0. Recipe resolution was checked through recipe_parser.load_workloads(scope='unit-tests', model='unit-tests', test_case='examples/multimodal_dev/tests/**/*.py', environment='dev', tag='latest'), which returns the workload plus its mcore-pyt-dev build dependency like every other bucket, and find_test_cases.py returns no --ignore args (no child buckets). Bucket wall clock ~40 s.

Issue tracking

Linked issue: none. This is a test/CI change with no runtime impact.

Contribution process

Pre-checks

• I have added relevant unit tests
• I have added relevant functional tests — n/a, this PR is test/CI wiring
• I have added proper typing to my code — matches the surrounding suite's style
• I have added relevant documentation — the recipe entry carries a comment explaining the dev/latest scoping, and each converted module documents its pytest invocation
• I have run the autoformatter.sh on my PR — a no-op here: it only considers .py files under megatron/core and tests/, and this PR's only tests/ change is a YAML

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[BestJuly]

/claude review

[claude]

Light review. The core of this looks right — I verified the claims that matter:

• The mrope_section=None / apply_rope_fusion=False stub fix is correct: _is_raw_mrope_freqs_thd short-circuits on config.mrope_section is None before the t.dim() != 3 / freqs.shape[1] != 1 raise branches (rope_utils.py:90-97), so the [3, 1, 1, 8] freqs tensor in that test — which is raw-looking — correctly stays off the raw-mRoPE path and lands on the monkeypatched _apply_rotary_pos_emb_thd.
• The recipe comment about GitHub CI ignoring the matrix is accurate: cicd-main.yml:680 parses only .products[].test_case[], and .github/actions/action.yml:137-141 hardcodes --environment dev with tag: latest from the caller.
• find_test_cases.py returns no --ignore args either direction — is_child_of_bucket compares base paths, and neither examples/multimodal_dev/tests nor tests/unit_tests is a prefix of the other. No overlap with the catch-all tests/unit_tests/**/*.py bucket.
• Routing every init/teardown through Utils keeps Utils.inited honest, and Utils.initialize_model_parallel/destroy_model_parallel pop NVTE_FLASH_ATTN/NVTE_FUSED_ATTN themselves — consistent with the conftest docstring's claim about deliberately not reproducing set_env.
• Grad hygiene in the shared thd_model module fixture is fine: run_equal_length_test zeroes between BSHD and THD and again at exit, so the smoke test starts clean and neither test is order-dependent.

Two non-blocking notes:

1. aligned_seq_len uses max where the docstring promises all (test_cp_correctness.py:232-240)

The docstring says it rounds up "to a multiple of 2*cp for every CP size given," but the implementation aligns only to 2 * max(cp_sizes). That is a multiple of every 2*cp only when max(cp_sizes) is divisible by each of the others. It holds for the current CP_SIZES = [2, 4], so nothing is broken today — but adding a non-dividing size (e.g. [2, 3, 4] → align = 8, which is not a multiple of 6) would silently produce a seq_len that fails zigzag splitting for one of the trials. LCM would make the code match the contract:

import math  # at top of module

def aligned_seq_len(seq_len, cp_sizes):
    align = 2 * math.lcm(*cp_sizes)
    return ((seq_len + align - 1) // align) * align

Alternatively just narrow the docstring to say it aligns to the largest CP size.

2. The repaired stub reproduces the drift it exists to catch (test_mrope_parity.py:667-676)

This is exactly the failure mode the PR is built to prevent: a hand-rolled SimpleNamespace that goes stale the next time rope_utils reads a new config attribute. The bucket will now catch the next occurrence rather than let it land silently, which is the main win — but constructing a real TransformerConfig here (no distributed init needed) would make the test immune instead of merely loud. Worth considering as a follow-up rather than a change to this PR.