Skip to content

Fix MIRACL eval leakage via dev split and add full English MTEB evaluation (bf16 + FlashAttention-2) - #37

Merged
iamtatsuki05 merged 11 commits into
developfrom
feature/v2-additional-experiments
Aug 3, 2026
Merged

Fix MIRACL eval leakage via dev split and add full English MTEB evaluation (bf16 + FlashAttention-2)#37
iamtatsuki05 merged 11 commits into
developfrom
feature/v2-additional-experiments

Conversation

@iamtatsuki05

@iamtatsuki05 iamtatsuki05 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

WHY

The alignment/uniformity and geometry analyses loaded miracl/miracl with a hard-coded split='train', which overlaps with the FT training data (ruri-v3-dataset-ft's miracl subset is built from the same train queries). In addition, the paper's English MTEB numbers covered only one task per family, so we re-run the full MTEB(eng, v2) task families under the same precision/attention setup used for training.

WHAT

MIRACL leak fix (alignment / uniformity)

  • Expose miracl_split (and positive-pair split) in the eval config and default all evaluation positive-pair splits to dev
  • Pass prepare_dataset arguments by keyword to avoid positional drift

Isotropic eval harness hardening

  • Add dtype / attn_implementation fields to CLIConfig (bfloat16 with flash_attention_2 or sdpa)
  • Load models with trust_remote_code so bidirectional remote-code models do not silently fall back to causal attention, and validate the reported attention backend / dtype
  • Enforce the 8192-token contract when the model supports it; measure short-context models at their native lengths with a warning
  • Add English base-model (pre-training only) isotropic configs

Full English MTEB evaluation

  • Add per-model full MTEB(eng, v2) configs (37 tasks across the 5 reported task families) and 120 chunked configs for parallel scheduling
  • Load MTEB eval models in bfloat16 (FlashAttention-2 rejects fp32)
  • Sanitize empty texts at the tokenize level (FiQA contains empty documents, which crash tokenizers that do not force special tokens; MTEB passes numpy arrays, so the sanitization hooks tokenize rather than encode)

Misc

  • Apply ruff format to the isotropic eval script and sentence-model config

The alignment/uniformity and geometry analyses loaded miracl/miracl with a
hard-coded split='train', which overlaps with the FT training data
(ruri-v3-dataset-ft's miracl subset is built from the same train queries).
Expose miracl_split in the eval config (default unchanged) so the
measurements can run on the dev split, and pass prepare_dataset arguments
by keyword to avoid positional drift.
@iamtatsuki05 iamtatsuki05 self-assigned this Aug 2, 2026
Evaluating on the train splits overlaps with the embedding training data
(MIRACL ja train via ruri-v3-dataset-ft, AllNLI train via the English FT
set), so make dev the default for miracl_split and
positive_pair_dataset_split and update the isotropic eval configs
accordingly.
…nfig

The current eval.py reads cfg.dtype and cfg.attn_implementation, but
CLIConfig never defined them, so every run crashed with AttributeError.
Add the fields and allow bfloat16+sdpa in addition to
bfloat16+flash_attention_2 for clusters without flash-attn.
@iamtatsuki05
iamtatsuki05 changed the base branch from main to develop August 2, 2026 15:00
- Load models with trust_remote_code so the bidirectional (-Bi) checkpoints
  use their custom attention instead of silently falling back to causal.
- Accept max_position_embeddings >= 8192 and raise max_seq_length to the
  8192 contract for long-context models; measure short-context models
  (2048) at their native lengths with a warning.
- Add isotropic configs for the English pre-trained-only baselines
  (ModernBERT-EN, Llama-Bi-EN, sarashina2.2-Bi measured on English data).
@iamtatsuki05 iamtatsuki05 changed the title Make the MIRACL split for positive-pair evaluation configurable Fix MIRACL eval leakage via dev split and add full English MTEB evaluation (bf16 + FlashAttention-2) Aug 2, 2026
@iamtatsuki05
iamtatsuki05 force-pushed the feature/v2-additional-experiments branch from 97850e1 to 0ea081e Compare August 2, 2026 22:14
@iamtatsuki05
iamtatsuki05 merged commit 7b6143a into develop Aug 3, 2026
2 checks passed
@iamtatsuki05 iamtatsuki05 mentioned this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant