FlagOS 2.1 — FlagScale v2.0.0

Release v2.0.0

Changes since v1.0.0

New Features

• Add --mg-fl-prefer argument for Megatron-LM-FL vendor selection (#1183) by @lihongyang1990

Bug Fixes

• Fix change log error (#1170) by @Caozhou1995
• Fix change log error (#1171) by @Caozhou1995
• Ascend serve deps (#1191) by @Darryl233
• Fix writer in training & engram allreduce mode (#1200) by @zhaoyinglia
• Follow up on the latest Megatron-LM-FL version in the MUSA test environment (#1224) by @Darryl233

CI/Infrastructure

• Add Ascend NPU and MetaX multi-chip test support (#1160) by @zihugithub
• Migrate FlagScale CI to FlagCICD infrastructure (#1187) by @zihugithub
• Migrate Ascend NPU config to FlagCICD platform (#1190) by @zihugithub
• Support Agent asistant in Workflow (#1192) by @Darryl233
• Add auto build and push CUDA Docker images to Harbor pipeline (#1143) by @zihugithub
• Increase claude max-turns (#1194) by @Darryl233
• Fix Harbor image build workflow for self-hosted runners (#1198) by @BrianPei
• Update coverage report & update megatron version of metax ci workflow (#1199) by @Darryl233
• Merge multiple chip pipelines into one (#1209) by @Darryl233

Other

• Cherry-pick to 1.0.0 (#1169) by @Caozhou1995
• Update readme (#1172) by @Caozhou1995
• Update readme (#1173) by @Caozhou1995
• Add link to the Discord social media platform (#1153) by @tengqm
• Update flagscale installation and add hydra-core dependacy (#1175) by @Caozhou1995
• Update link to Facebook community (#1174) by @tengqm
• [inference] qwen_gr00t supports ascend and musa platform (#1178) by @1016qqz
• Add flagscale eval robo CLI (#1181) by @legitnull
• Deepseek Engram optimization. (#1147) by @LiJunscs
• FlagScale Train Upgrade: Synchronization with Megatron-LM Core0.17.0 (#1193) by @lxd-cumt
• : fix the bug of saving moe checkpoint when tp > ep * etp. (#1204) by @LiJunscs
• Add Qwen35 Model (#1196) by @wenqingqian
• Deepseek v4 Support (#1195) by @LiJunscs
• Fix DeepSeek-V3 checkpoint export compatibility (#1202) by @lihongyang1990
• Fix dualpipev (#1207) by @zhaoyinglia
• Override Upgrade (#1214) by @lxd-cumt
• Fix unittest (#1222) by @zhaoyinglia
• Cherry-Pick (#1223) by @zhaoyinglia
• Add perf monitor integration (#1216) by @lzy-edu
• Straggler detection rebased (#1215) by @lzy-edu

Contributors

Thanks to all contributors who made this release possible:

• @Caozhou1995 (6 PRs)
• @Darryl233 (6 PRs)
• @zhaoyinglia (4 PRs)
• @zihugithub (4 PRs)
• @LiJunscs (3 PRs)
• @lihongyang1990 (2 PRs)
• @lxd-cumt (2 PRs)
• @lzy-edu (2 PRs)
• @tengqm (2 PRs)
• @1016qqz
• @BrianPei
• @legitnull
• @wenqingqian

---

Generated by Release Notes Generator

v1.0.0

What's Changed

Highlights

Major updates: Introduced a unified FlagScale CLI as the single entry point for all operations,
with unified multi-chip training support across NVIDIA GPU, Ascend, and MUSA.
Replaced third-party verl with VeRL-FL and expanded model support
to include Qwen3-VL, Qwen2.5-VL, GR00T N1.5, and DeepSeek Engram.
Enhanced CI/CD coverage with Megatron-LM-FL integration tests and automated CLI validation workflows.

All Changes

• [CICD] Temporarily toggle off the rl test by @aoyulong in #1034
• Add README for llama.cpp by @Hchnr in #1030
• Update project README by @tengqm in #1033
• Move megatron-related codes from flagscale/train to flagscale/train/megatron by @lxd-cumt in #1037
• Add maintainers list to the project by @tengqm in #1038
• Refactor serve config yaml by @cyber-pioneer in #1031
• Refactor the code and rm accelerate dependency by @legitnull in #1032
• [Docs] Update quick start guide for better clarity by @zihugithub in #1041
• [Train] Fix device_type attr bug by @cyber-pioneer in #1040
• Adaptation of Qwen3-VL and Qwen2.5-VL by @Hchnr in #1039
• Update doc for PI0/PI0.5 by @legitnull in #1045
• Replace third_party/verl with verl-FL by @WenLiuyi in #1036
• [DOC] Add a document for RL by @heavyrain-lzy in #1047
• Update FlagOS-Robo related docs by @Hchnr in #1029
• Fix links in robo README by @Hchnr in #1052
• [CICD] Update workflows by @zihugithub in #1043
• [Docs] Notice users for the major refactoring by @aoyulong in #1053
• Refactor github workflows by @aoyulong in #1042
• [Serve] Refactor cloud launcher by @luoyc123 in #1048
• [CodeStyle] Update FS codestyle from black to ruff by @zhaoyinglia in #1055
• [Serve] Bugfix serve_id in YAML file can be any value by @luoyc123 in #1057
• add gpu_health_check for communication test by @wanglei19991004 in #1051
• Update docs for Qwen2.5-VL, RoboBrain-X0, RoboBrain-X0.5 by @Hchnr in #1060
• [Model] Update the document for training Qwen3-VL by @heavyrain-lzy in #1059
• Update RoboBrain2.5 Docs by @Hchnr in #1067
• [Train] Parse yaml config directly to configs without manually writing cli args by @legitnull in #1061
• [Docs] Update Qwen training dataset demo by @Darryl233 in #1065
• Update README using GitHub markdown syntax for announcement by @tengqm in #1072
• Refactor tests impls and github workflows by @aoyulong in #1066
• Remove dockerfile temporarily by @aoyulong in #1074
• Remove robotics backend by @Hchnr in #1070
• Move date preprocessing into Energon for RoboBrain-X0 training. by @Hchnr in #1069
• add gpu_health_check for computation check by @wanglei19991004 in #1064
• Improve the workflows and refactor the install process by @aoyulong in #1073
• Update links in RoboBrain2.5 docs by @Hchnr in #1083
• [Serve] Migrate multiple model function to new runner by @luoyc123 in #1062
• Support RoboBrain-X0 training with lerobot dataset by @Hchnr in #1082
• update runtime_env in rl configs by @physics31415926 in #1089
• [BugFix] Resume partial tests for inference and serving by @zihugithub in #1054
• Update project banner by @tengqm in #1096
• Refactor install scripts and add dockerfiles by @aoyulong in #1100
• [CICD] Update aquila tp2-pp2 ci for evaluation by @lxd-cumt in #1110
• Refactor pip install by @aoyulong in #1109
• Set logger stacklevel by @legitnull in #1113
• [serve] Fix deploy check by @cyber-pioneer in #1116
• Fix incorrect log path in robobrain2_5 README by @Teeeio in #1122
• [BugFix] Fix nesting checkpoint output dir by @legitnull in #1124
• [Train] Add optim_setup utilities and extend train_config with freeze/param_groups support by @legitnull in #1125
• Improve the installation and cli experience by @aoyulong in #1118
• add rl example config by @physics31415926 in #1114
• [CICD] replace souce install with whl install by @Darryl233 in #1121
• [fix] run.py relative path resolve by @Darryl233 in #1128
• [Docs] Update README to use FlagScale CLI commands by @zihugithub in #1130
• [CICD] refactor: migrate test runner from python run.py to flagscale CLI by @zihugithub in #1131
• [Tools] Support ascend vllm serve install by @Darryl233 in #1132
• Fix cicd evaluation by @lxd-cumt in #1139
• [CICD] Fix hetero-train functional tests error by @heavyrain-lzy in #1138
• Add links to social media by @tengqm in #1135
• [BugFix] Adjust robotics dependencies and vllm install description by @1016qqz in #1117
• Add the background and foreground mode for runners by @aoyulong in #1134
• [Train] Add DeepSeek Engram by @lxd-cumt in #1107
• [CICD] Support megatron-fl-lm integration test by @Darryl233 in #1142
• [Model] Add Qwen-GR00T VLA model by @legitnull in #1120
• Unified multiple platform support for flagscale training by @lxd-cumt in #1146
• Support nsys profile for flagscale train by @lxd-cumt in #1154
• [CICD] add CLI validation workflow for FlagScale CLI by @zihugithub in #1144
• [Benchmark] Add benchmark functional test framework with Qwen3 test case by @zihugithub in #1145
• [Model] Support Model GR00T N1.5 by @legitnull in #1151
• [FIx] fix serve functional test connection 503 by @Darryl233 in #1163
• [CICD] add coverage report for CICD by @Darryl233 in #1159
• pi0/pi0.5/groot support ascend/musa and platform implement by @1016qqz in #1162
• [Train] Support FSDP2 checkpoint resume by @legitnull in #1149
• [CherryPick] Cherry-pick to 1.0.0 by @Caozhou1995 in #1169

New Contributors

• @WenLiuyi made their first contribution in #1036
• @physics31415926 made their first contribution in #1089
• @1016qqz made their first contribution in #1117

Full Changelog: v1.0.0-alpha.0...v1.0.0

v1.0.0-alpha.0

• Major updates: Refactored the codebase by moving hardware-specific (multi-chip) support into plugin repositories such as TransformerEngine-FL and vllm-plugin-FL. These plugins build on top of FlagOS, a unified open-source AI system software stack. We also re-initialized the Git commit history to reduce repository size.
• Compatibility (legacy users): If you are using or upgrading from a version earlier than v1.0.0-alpha.0, please use the main-legacy branch. It will continue to receive critical bug fixes and minor updates for a period of time.

v0.9.0

• Training & Finetuning: Added LoRA for efficient finetuning, improved the autotuner for cross-chip heterogeneous training, and enabled distributed RWKV training.
• Inference & Serving: Introduced DiffusionEngine for FLUX.1-dev, Qwen-Image, and Wan2.1-T2V, support multi-model automatic orchestration and dynamic scaling.
• Embodied AI: Full lifecycle support for Robobrain, Robotics, and PI0, plus semantic retrieval for MCP-based skills for RoboOS.
• Elastic & Fault Tolerance: Detect task status automatically (errors, hangs, etc.) and periodically record them.
• Hardware & System: Broader chip support, upgraded patch mechanism with file-level diffs, and enhanced CICD for different chips.

v0.8.0

• Introduced a new flexible and robust multi-backend mechanism and updated vendor adaptation methods.
• Enabled heterogeneous prefill-decoding disaggregation across vendor chips within a single instance via FlagCX (beta).
• Upgraded DeepSeek-v3 pre-training with the new Megatron-LM and added heterogeneous pre-training across different chips for MoE models like DeepSeek-v3.

v0.6.5

• Added support for DeepSeek-V3 distributed pre-training (beta) and DeepSeek-V3/R1 serving across multiple chips.
• Introduced an auto-tuning feature for serving and a new CLI feature for one-click deployment.
• Enhanced the CI/CD system to support more chips and integrated the workflow of FlagRelease.

v0.6.0

• Introduced general multi-dimensional heterogeneous parallelism and CPU-based communication between different chips.
• Added comprehensive support for data processing and faster distributed training of LLaVA-OneVision, achieving SOTA results on the Infinity-MM dataset.
• Open-sourced the optimized CFG implementation and accelerated the generation and understanding tasks for Emu3.
• Implemented the auto-tuning feature to simplify large-scale distributed training, making it more accessible for users with less expertise.
• Enhanced the CI/CD system to facilitate more efficient unit testing across different backends and perform the loss check for the various parallel strategies.

v0.3

• Accomplish the heterogeneous hybrid training of the Aquila2-70B-Expr model on a cluster utilizing a combination of NVIDIA and Iluvatar chips.
• Provide the training of the Aquila2 series across a variety of AI chips from six distinct manufacturers.

v0.2

• Provide the actually used training scheme for Aquila2-70B-Expr, including the parallel strategies, optimizations and hyper-parameter settings.
• Support heterogeneous training on chips of different generations with the same architecture or compatible architectures, including NVIDIA GPUs and Iluvatar CoreX chips.
• Support training on chinese domestic hardwares, including Iluvatar CoreX and Baidu KUNLUN chips.