Extend v2 experiments: English bidirectional models, GLUE/MTEB/QA evaluation, and training-free geometry analysis - #35
Merged
Conversation
Develop to Main
Sync Develop to Main
Sync: Develop to Main
Sync: Develop to Main
- Created JSON configuration files for Sentence-Llama-Bi-3B, Sentence-ModernBERT-3B, and Sentence-Sarashina-Bi-3B models for benchmarking with the wiki40b_ja dataset. - Added isotropic evaluation configurations for Llama-3B, Llama-Bi-3B, ModernBERT-3B, and Sarashina-3B models, specifying parameters for evaluation. - Introduced PT evaluation configurations for Llama-Bi-JP-3B and Llama-JP-3B models across JCoLA, JNLI, and JSTS tasks, including training and evaluation settings. - Ensured all configurations include necessary parameters such as model paths, output directories, and training hyperparameters.
…aining configurations
- Added support for JGLUE tasks in the configuration, including a new function to retrieve benchmark dataset specifications. - Introduced dataset revision parameter in DataTrainingArguments for better dataset version control. - Expanded CLIConfig in sentence model to include parameters for positive sentence pairs datasets. - Updated dataset preparation logic to handle positive pairs from specified datasets. - Added unit tests for JGLUE configuration to ensure correct dataset retrieval and error handling. - Updated dependencies in the project lock file to include new packages and versions.
…ModernBERT models
…ning configurations
…d Japanese - Introduced new JSON configuration files for Llama and ModernBERT models, including fine-tuning and pre-training setups for both English and Japanese. - Updated README files to reflect the addition of new configurations for Llama-Bi-EN, ModernBERT-EN, and their respective training stages. - Included detailed configurations for training and fine-tuning processes, specifying parameters such as learning rates, batch sizes, and dataset paths.
…ate validation_split_percentage for Llama and ModernBERT models - Added "ddp_timeout": 86400 to training configurations for Llama and ModernBERT models in both English and Japanese. - Updated validation_split_percentage to 10 in Llama-EN-3B-PT-stage1.json and ModernBERT-EN-3B-PT-stage1.json. - Increased num_train_epochs to 6 in Llama-JP-3B and ModernBERT-JP-3B training configurations. - Set streaming to false in training configuration files for Llama and ModernBERT models.
- Increased gradient accumulation steps for Llama-EN-3B-PT-stage1 and Llama-JP-3B-PT-stage1 from 2048 to 4096. - Increased gradient accumulation steps for Llama-EN-3B-PT-stage2 and Llama-JP-3B-PT-stage2 from 512 to 1024. - Increased gradient accumulation steps for Llama-EN-1B-PT-stage2 and Llama-JP-1B-PT-stage2 from 512 to 1024. - Added new configuration files for Llama-JP-1B and Llama-EN-1B for both stage 1 and stage 2 pre-training. - Added new configuration files for ModernBERT-JP-1B and ModernBERT-EN-1B for both stage 1 and stage 2 pre-training. - Updated README files to include new model configurations and examples. - Enhanced model initialization script to allow optional tokenizer path.
…s training configurations
…requency - Changed "logging_steps" from 10 to 1 in multiple training configuration files for various models, including Sentence-Llama, ModernBERT, and Sarashina, to enhance the granularity of logging during training.
… ModernBERT training configurations
…and ModernBERT training configurations
- Changed wandb version from >=0.19.10 to ==0.20.1 in pyproject.toml and uv.lock. - Added setproctitle package with version 1.3.7 to uv.lock.
…ng configurations across multiple models
…ss multiple models
…ng configurations
…e dependencies - Added "gather_across_devices": true to various model configuration JSON files to enable gathering embeddings across devices before computing cached loss. - Updated the sentence-transformers dependency version from 4.1.0 to 5.1.0 in pyproject.toml and uv.lock. - Modified run_st.py to pass the gather_across_devices argument to the CachedMultipleNegativesRankingLoss. - Introduced gather_across_devices field in ModelArguments class for better control over the gathering behavior.
…dient accumulation steps for improved performance
…dient accumulation steps for multiple models
…dient accumulation steps for multiple models
…ization - Introduced `modeling_bidirectional_qwen2.py` with `Qwen2BiModel` and `Qwen2BiForMNTP` classes for bidirectional processing. - Enhanced `initialize_model` function to support tokenizer revisions and repository IDs. - Updated `save_custom_model` to handle new Qwen2 model classes and configurations. - Modified `DataTrainingArguments` in both CLM and MLM training scripts to include paths for tokenized datasets.
…cross various tasks and datasets, including RTE, SST-2, MRPC, QNLI, QQP, and JNLI. This cleanup enhances project maintainability by eliminating unused or redundant files.
iamtatsuki05
changed the title
…structure and add detailed JSON output descriptions; refactor evaluation metrics in eval.py to include squared L2 distances and histograms; implement new metric functions in metric.py and corresponding tests in test_metric.py.
…across COLA, MNLI, and STSB tasks
…pdate evaluation scripts to utilize batch size for encoding
…rt bfloat16 and flash_attention_2 - Changed batch_size from 8 to 2 in multiple model configuration files for isotropic evaluation. - Added dtype as bfloat16 and attn_implementation as flash_attention_2 to all relevant JSON configuration files. - Enhanced eval.py to validate model loading with specified dtype and attention implementation, ensuring compatibility with the new settings.
…mance - Increased learning rates from 1e-5 to 2e-5 across various models. - Reduced batch sizes from 4 to 2 for training and evaluation to optimize resource usage. - Decreased gradient accumulation steps from 128 to 16 for faster training iterations. - Enabled bf16 and bf16_full_eval for improved performance on supported hardware. - Introduced flash attention implementation for better memory efficiency. - Set deepspeed configurations to null to simplify training setup.
…s tasks - Created JSON configuration files for Sarashina-EN and Sarashina-JP models for tasks including STSB, CoLA, MNLI, JCoLA, JNLI, and JSTS. - Configurations include model paths, training parameters, evaluation strategies, and logging settings. - Ensured compatibility with the GLUE benchmark datasets and specified hyperparameters for training and evaluation.
- Set "deepspeed" to "./config/constract_llm/train/ds_config/ds_zero2.json" in various evaluation configuration files for ModernBERT and Sarashina models, ensuring consistent usage of DeepSpeed across different tasks (CoLA, MNLI, STSB). - Adjust output and logging directories for ModernBERT-JP-0.5B-PT-stage2_jcola.json to point to the new evaluation output structure.
- Adjusted learning rates for ModernBERT-JP-3B-PT-stage2, Qwen2.5-EN-0.5B, Sarashina-EN-0.5B, Sarashina-EN-1B, and Sarashina-EN-3B to improve training stability. - Changed metric for best model from "eval_loss" to "accuracy" for classification tasks and "pearson" for regression tasks across multiple configurations. - Reduced save and evaluation steps from 500 to 100 for quicker feedback during training. - Enabled loading the best model at the end of training and set "greater_is_better" to true for appropriate metrics.
- Implemented `run_qa.py` for fine-tuning models on extractive QA tasks (SQuAD/JSQuAD). - Created `QuestionAnsweringTrainer` class for specialized training and evaluation logic. - Added utility functions for post-processing QA predictions in `utils_qa.py`. - Introduced data classes for handling training arguments and dataset configurations. - Established logging and error handling for non-finite metrics during evaluation. - Included support for beam search in post-processing predictions. - Updated module structure with new files and directories for QA evaluation.
…s to optimize training efficiency.
…an for improved accuracy
…embedding checkpoints
iamtatsuki05
changed the title
…umps to the analysis runner
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
Extend v2 experiments to English models (international extension), and add the training-free analysis suite used for the mechanistic study of encoder vs decoder sentence embeddings.
WHAT
Models / training
Evaluation
Training-free geometry / attention analysis suite (scripts/constract_llm/eval/analysis/)
VERIFICATION