You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
🔥🔥 [2024/01/17] We released MFTCoder v0.3.0, mainly for MFTCoder-accelerate. It now supports new models like Mixtral(MoE), DeepSeek-coder, chatglm3. It supports FSDP as an option. It also supports Self-paced Loss as a solution for convergence balance in Multitask Fine-tuning.
49
+
50
+
🔥🔥 [2024/01/17][CodeFuse-DeepSeek-33B](https://huggingface.co/codefuse-ai/CodeFuse-DeepSeek-33B) has been released, achieving a pass@1 (greedy decoding) score of 78.7% on HumanEval. It achieves top1 win-rate on Bigcode Leardboard.
51
+
52
+
🔥🔥 [2024/01/17][CodeFuse-Mixtral-8x7B](https://huggingface.co/codefuse-ai/CodeFuse-Mixtral-8X7B) has been released, achieving a pass@1 (greedy decoding) score of 56.1% on HumanEval.
53
+
44
54
🔥🔥 [2023/11/07][MFTCoder Paper](https://arxiv.org/abs/2311.02303) has been released on Arxiv, which discloses technique details of multi-task-fine-tuning.
45
55
46
56
🔥🔥 [2023/10/20][CodeFuse-QWen-14B](https://huggingface.co/codefuse-ai/CodeFuse-QWen-14B) has been released, achieving a pass@1 (greedy decoding) score of 48.8% on HumanEval, which gains 16% absolute improvement over the base model [Qwen-14b](https://huggingface.co/Qwen/Qwen-14B)
**High Accuracy and efficiency multi-task fine-tuning framework for Code LLMs.**
91
+
**High Accuracy and efficiency Multi-task Fine-tuning framework for Code LLMs.**
92
+
93
+
**CodeFuse-MFTCoder** is an open-source project of CodeFuse for accurate and efficient Multi-task Fine-tuning(MFT) on Large Language Models(LLMs), especially on Code-LLMs(large language model for code tasks).
94
+
Moreover, we open source Code LLM models and code-related datasets along with the MFTCoder framework.
80
95
81
-
**CodeFuse-MFTCoder** is an open-source project of CodeFuse for multitasking Code-LLMs(large language model for code tasks), which includes models, datasets, training codebases and inference guides.
82
96
In MFTCoder, we released two codebases for finetuning Large Language Models:
83
-
-```mft_peft_hf``` is based on the HuggingFace Accelerate and deepspeed framework.
84
-
-```mft_atorch``` is based on the [ATorch frameworks](https://github.com/intelligent-machine-learning/dlrover), which is a fast distributed training framework of LLM.
97
+
-```MFTCoder-accelerate``` is a framework with accelerate and DeepSpeed/FSDP. All tech-stacks are open-source and vibrant. We highly recommend you try this framework and make your fintuning accurate and efficient.
98
+
-```MFTCoder-atorch``` is based on the [ATorch frameworks](https://github.com/intelligent-machine-learning/dlrover), which is a fast distributed training framework of LLM.
85
99
86
100
The aim of this project is to foster collaboration and share advancements in large language models, particularly within the domain of code development.
87
101
88
102
### Frameworks
89
-
103
+
90
104
91
105
### Highlights
92
106
:white_check_mark:**Multi-task**: Train models on multiple tasks while maintaining a balance between them. The models can even generalize to new, previously unseen tasks.
93
107
94
108
:white_check_mark:**Multi-model**: It integrates state-of-the-art open-source models such as gpt-neox, llama, llama-2, baichuan, Qwen, chatglm2, and more. (These finetuned models will be released in the near future.)
95
109
96
-
:white_check_mark:**Multi-framework**: It provides support for both HuggingFace Accelerate (with deepspeed) and [ATorch](https://github.com/intelligent-machine-learning/dlrover).
110
+
:white_check_mark:**Multi-framework**: It provides support for both Accelerate (with Deepspeed and FSDP) and ATorch
97
111
98
-
:white_check_mark:**Efficient fine-tuning**: It supports LoRA and QLoRA, enabling fine-tuning of large models with minimal resources. The training speed meets the demands of almost all fine-tuning scenarios.
112
+
:white_check_mark:**Efficient fine-tuning**: It supports LoRA, QLoRA as well as Full-parameters training, enabling fine-tuning of large models with minimal resources. The training speed meets the demands of almost all fine-tuning scenarios.
99
113
100
114
The main components of this project include:
101
115
- Support for both SFT (Supervised FineTuning) and MFT (Multi-task FineTuning). The current MFTCoder achieves data balance among multiple tasks, and future releases will achieve a balance between task difficulty and convergence speed during training.
102
-
- Support for QLoRA instruction fine-tuning, as well as LoRA fine-tuning.
103
-
- Support for most mainstream open-source large models, particularly those relevant to Code-LLMs, such as Code-LLaMA, Starcoder, Codegeex2, Qwen, GPT-Neox, and more.
116
+
- Support for QLoRA instruction fine-tuning, LoRA fine-tuning as well as Full-parameters fine-tuning.
117
+
- Support for most mainstream open-source large models, particularly those relevant to Code-LLMs, such as DeepSeek-coder, Mistral, Mixtral, Chatglm3, Code-LLaMA, Starcoder, Codegeex2, Qwen, GPT-Neox, and more.
104
118
- Support for weight merging between the LoRA adaptor and base models, simplifying the inference process.
105
119
- Release of 2 high-quality code-related instruction fine-tuning datasets: [Evol-instruction-66k](https://huggingface.co/datasets/codefuse-ai/Evol-instruction-66k) and [CodeExercise-Python-27k](https://huggingface.co/datasets/codefuse-ai/CodeExercise-Python-27k).
106
-
- Release of 2 models: [CodeFuse-13B](https://huggingface.co/codefuse-ai/CodeFuse-13B) and [CodeFuse-CodeLlama-34B](https://huggingface.co/codefuse-ai/CodeFuse-CodeLlama-34B).
120
+
- Release of many Code LLMs, please refer to organizations: [codefuse-ai on huggingface](https://huggingface.co/codefuse-ai) or [codefuse-ai on modelscope](https://modelscope.cn/organization/codefuse-ai).
107
121
108
122
109
123
## Requirements
@@ -113,28 +127,36 @@ Next, we have provided an init_env.sh script to simplify the installation of req
113
127
```bash
114
128
sh init_env.sh
115
129
```
116
-
If you require flash attention, please refer to the following link for installation instructions: https://github.com/Dao-AILab/flash-attention
130
+
We highly recommend training with flash attention(version >= 2.1.0, preferably 2.3.6), please refer to the following link for installation instructions: https://github.com/Dao-AILab/flash-attention
117
131
118
132
119
133
## Training
120
-
🚀 [Huggingface accelerate + deepspeed Codebase for MFT(Multi-task Finetuning)](./mft_peft_hf/README.md)
134
+
As mentioned above, we open source two training frameworks. You could refer to their own READMEs for more details as followed.
121
135
122
-
🚀 [Atorch Codebase for MFT(Multi-task Finetuning)](./mft_atorch/README.md)
136
+
If you are familiar with open source ```transformers```, ```DeepSpeed``` or ```FSDP```, we highly recommend you try:
123
137
138
+
🚀🚀 [MFTCoder-accelerate: Accelerate + Deepspeed/FSDP Codebase for MFT(Multi-task Finetuning)](mftcoder_accelerate/README.md)
124
139
125
-
## Models
126
140
127
-
We are excited to release the following two CodeLLMs trained by MFTCoder, now available on Hugging Face:
141
+
If you want to explore some new framework like atorch, you could check:
142
+
143
+
🚀 [MFTCoder-atorch: Atorch Codebase for MFT(Multi-task Finetuning)](mftcoder_atorch/README.md)
144
+
145
+
146
+
## Models
128
147
148
+
We are excited to release the following two CodeLLMs trained by MFTCoder, now available on both HuggingFace and ModelScope:
129
149
130
-
| Model | Base Model | Num of examples trained | Batch Size | Seq Length |
0 commit comments