Sync: Develop to Main - #38
Merged
Merged
Conversation
The alignment/uniformity and geometry analyses loaded miracl/miracl with a hard-coded split='train', which overlaps with the FT training data (ruri-v3-dataset-ft's miracl subset is built from the same train queries). Expose miracl_split in the eval config (default unchanged) so the measurements can run on the dev split, and pass prepare_dataset arguments by keyword to avoid positional drift.
Evaluating on the train splits overlaps with the embedding training data (MIRACL ja train via ruri-v3-dataset-ft, AllNLI train via the English FT set), so make dev the default for miracl_split and positive_pair_dataset_split and update the isotropic eval configs accordingly.
…nfig The current eval.py reads cfg.dtype and cfg.attn_implementation, but CLIConfig never defined them, so every run crashed with AttributeError. Add the fields and allow bfloat16+sdpa in addition to bfloat16+flash_attention_2 for clusters without flash-attn.
- Load models with trust_remote_code so the bidirectional (-Bi) checkpoints use their custom attention instead of silently falling back to causal. - Accept max_position_embeddings >= 8192 and raise max_seq_length to the 8192 contract for long-context models; measure short-context models (2048) at their native lengths with a warning. - Add isotropic configs for the English pre-trained-only baselines (ModernBERT-EN, Llama-Bi-EN, sarashina2.2-Bi measured on English data).
…e five task families)
…eval-task chunks plus two light bundles)
…ts crash tokenizers without forced special tokens)
… bypassing list checks in encode)
…ips whitespace inside Transformer.tokenize)
…ments Fix MIRACL eval leakage via dev split and add full English MTEB evaluation (bf16 + FlashAttention-2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
WHAT