Performance decrease after finetuning on 8*A800 GPUs. Got 0.367 in aime24_nofigures task

[LiZhangMing]

Here is my test code：
CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3 lm_eval --model vllm --model_args pretrained=/home/lizhangming/lzm_project/s1/ckpts/s1_20250625_075920,tokenizer=/home/lizhangming/lzm_project/s1/ckpts/s1_20250625_075920,dtype=float32,tensor_parallel_size=8 --tasks aime24_nofigures --batch_size auto --apply_chat_template --output_path results/aime24_nofigures_full_fintune --log_samples --gen_kwargs "max_gen_toks=32768,max_tokens_thinking=auto,thinking_n_ignore=2,thinking_n_ignore_str=Wait"

I use the deepspeed_zero3 to finetuning Qwen/Qwen2.5-32B-Instruct.
here is my code:
uid="$(date +%Y%m%d_%H%M%S)"
base_model="Qwen/Qwen2.5-32B-Instruct"
lr=1e-5
epochs=5
micro_batch_size=2
push_to_hub=false
gradient_accumulation_steps=1
max_steps=-1
gpu_count=8

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file train/deepspeed_zero3.yaml --deepspeed_config_file train/ds_config.json
train/sft.py
--per_device_train_batch_size=${micro_batch_size}
--per_device_eval_batch_size=${micro_batch_size}
--gradient_accumulation_steps=${gradient_accumulation_steps}
--num_train_epochs=${epochs}
--max_steps=${max_steps}
--train_file_path="simplescaling/s1K_tokenized"
--model_name=${base_model}
--warmup_ratio=0.05
--bf16=True
--eval_strategy="steps"
--eval_steps=50
--logging_steps=1
--lr_scheduler_type="cosine"
--learning_rate=${lr}
--weight_decay=1e-4
--adam_beta1=0.9
--adam_beta2=0.95
--output_dir="ckpts/s1_${uid}"
--save_only_model=True
--gradient_checkpointing=True
--save_strategy=no
--dataset_text_field="text"

deepspeed_zero3.yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

ds_config.json
{
"train_micro_batch_size_per_gpu": "auto",
"gradient_accumulation_steps": 1,
"gradient_clipping": 1.0,
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "none"
},
"stage3_param_persistence_threshold": 1e6,
"stage3_max_live_parameters": 1e9,
"stage3_prefetch_bucket_size": 5e7
},
"bf16": {
"enabled": true
}
}

[Muennighoff]

Can you try finetuning on https://huggingface.co/datasets/simplescaling/s1K-1.1_tokenized instead?

[LiZhangMing]

Thanks your advice, Iwill try it.  But, Could you give me some advices  to realize your result in s1k.

…

---Original--- From: "Niklas ***@***.***> Date: Thu, Jun 26, 2025 22:55 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [simplescaling/s1] Performance decrease after finetuning on8*A800 GPUs. Got 0.367 in aime24_nofigures task (Issue #123) Muennighoff left a comment (simplescaling/s1#123) Can you try finetuning on https://huggingface.co/datasets/simplescaling/s1K-1.1_tokenized instead? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jnward]

Hm, I just did the same thing with 8xH200 (using gradient accumulation instead of increasing microbatch size) and got exactly the same result on AIME. I've had better results with s1k-1.1 but was hoping to use the original dataset since it's much faster to train (and I'm running a few experiments)!

Not sure if there are better hyperparams to use? Training loss looks kinda overfit

[Muennighoff]

I'd use the hparams in this repo esp. block size is important

[jnward]

The only ones I changed are gradient accumulation (which shouldn't affect anything afaict since I'm using the same effective batch size) and lowered block size to 20k... but I don't think there are very many (any?) samples in the s1k dataset above 20k tokens and I'm not using budget forcing for the evals

[LiZhangMing]

The only ones I changed are gradient accumulation (which shouldn't affect anything afaict since I'm using the same effective batch size) and lowered block size to 20k... but I don't think there are very many (any?) samples in the s1k dataset above 20k tokens and I'm not using budget forcing for the evals

that is interesting， could you give me your evals code, like 'CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3 lm_eval --model vllm --model_args pretrained=/home/lizhangming/lzm_project/s1/ckpts/s1_20250625_075920,tokenizer=/home/lizhangming/lzm_project/s1/ckpts/s1_20250625_075920,dtype=float32,tensor_parallel_size=8 --tasks aime24_nofigures --batch_size auto --apply_chat_template --output_path results/aime24_nofigures_full_fintune --log_samples --gen_kwargs "max_gen_toks=32768,max_tokens_thinking=auto,thinking_n_ignore=2,thinking_n_ignore_str=Wait"'.

[jnward]

could you give me your evals code

I'm using this bash script:

export NCCL_NVLS_ENABLE=0
export NCCL_ALGO=Ring
export NCCL_DEBUG=WARN
export CUDA_LAUNCH_BLOCKING=0

MODEL_PATH=$1
if [ -z "$MODEL_PATH" ]; then
    echo "Usage: bash eval.sh <model_path>"
    exit 1
fi

# Strip trailing slashes from adapter path
MODEL_PATH=${MODEL_PATH%/}

MODEL_SIZE=32B

cd eval/lm-evaluation-harness

PROCESSOR=gpt-4o-mini lm_eval \
    --model vllm \
    --model_args "pretrained=${PWD}/../../${MODEL_PATH},dtype=bfloat16,tensor_parallel_size=2" \
    --tasks aime24_nofigures,openai_math \
    --batch_size auto \
    --apply_chat_template \
    --output_path "../../results/${MODEL_SIZE}_ft" \
    --log_samples \
    --gen_kwargs "max_gen_toks=20000"

cd ../..

echo "Evaluation complete! Results saved to: results/${MODEL_SIZE}_ft"

also here is my training config:

# Reference Running: bash train/sft.sh
# {'train_runtime': 5268.8407, 'train_samples_per_second': 0.949, 'train_steps_per_second': 0.119, 'train_loss': 0.1172730620391667, 'epoch': 5.0}
uid="$(date +%Y%m%d_%H%M%S)"
base_model="Qwen/Qwen2.5-32B-Instruct"
block_size=20000
lr=1e-5
min_lr=0
epochs=5
weight_decay=1e-4 # -> the same training pipe as slurm_training
batch_size=16
micro_batch_size=1
max_steps=-1
gpu_count=$(nvidia-smi -L | wc -l)
gradient_accumulation_steps=$((batch_size / (micro_batch_size * gpu_count)))
push_to_hub=false

torchrun --nproc-per-node ${gpu_count} --master_port 12345 \
    train/sft.py \
    --block_size=${block_size} \
    --per_device_train_batch_size=${micro_batch_size} \
    --per_device_eval_batch_size=${micro_batch_size} \
    --gradient_accumulation_steps=${gradient_accumulation_steps} \
    --num_train_epochs=${epochs} \
    --train_file_path="simplescaling/s1K_tokenized" \
    --model_name=${base_model} \
    --warmup_ratio=0.05 \
    --fsdp="full_shard auto_wrap" \
    --fsdp_config="train/fsdp_config_qwen.json" \
    --bf16=True \
    --eval_strategy="no" \
    --logging_steps=1 \
    --save_strategy="no" \
    --lr_scheduler_type="cosine" \
    --learning_rate=${lr} \
    --weight_decay=${weight_decay} \
    --adam_beta1=0.9 \
    --adam_beta2=0.95 \
    --output_dir="ckpts/s1-${uid}" \
    --push_to_hub=${push_to_hub} \
    --save_only_model=True \
    --gradient_checkpointing=True
    # --accelerator_config='{"gradient_accumulation_kwargs": {"sync_each_batch": true}}'

[LiZhangMing]

could you give me your evals code

I'm using this bash script:

export NCCL_NVLS_ENABLE=0
export NCCL_ALGO=Ring
export NCCL_DEBUG=WARN
export CUDA_LAUNCH_BLOCKING=0

MODEL_PATH=$1
if [ -z "$MODEL_PATH" ]; then
echo "Usage: bash eval.sh <model_path>"
exit 1
fi

Strip trailing slashes from adapter path

MODEL_PATH=${MODEL_PATH%/}

MODEL_SIZE=32B

cd eval/lm-evaluation-harness

PROCESSOR=gpt-4o-mini lm_eval
--model vllm
--model_args "pretrained=${PWD}/../../${MODEL_PATH},dtype=bfloat16,tensor_parallel_size=2"
--tasks aime24_nofigures,openai_math
--batch_size auto
--apply_chat_template
--output_path "../../results/${MODEL_SIZE}_ft"
--log_samples
--gen_kwargs "max_gen_toks=20000"

cd ../..

echo "Evaluation complete! Results saved to: results/${MODEL_SIZE}_ft"
also here is my training config:

Reference Running: bash train/sft.sh

{'train_runtime': 5268.8407, 'train_samples_per_second': 0.949, 'train_steps_per_second': 0.119, 'train_loss': 0.1172730620391667, 'epoch': 5.0}

uid="$(date +%Y%m%d_%H%M%S)"
base_model="Qwen/Qwen2.5-32B-Instruct"
block_size=20000
lr=1e-5
min_lr=0
epochs=5
weight_decay=1e-4 # -> the same training pipe as slurm_training
batch_size=16
micro_batch_size=1
max_steps=-1
gpu_count=$(nvidia-smi -L | wc -l)
gradient_accumulation_steps=$((batch_size / (micro_batch_size * gpu_count)))
push_to_hub=false

torchrun --nproc-per-node ${gpu_count} --master_port 12345
train/sft.py
--block_size=${block_size}
--per_device_train_batch_size=${micro_batch_size}
--per_device_eval_batch_size=${micro_batch_size}
--gradient_accumulation_steps=${gradient_accumulation_steps}
--num_train_epochs=${epochs}
--train_file_path="simplescaling/s1K_tokenized"
--model_name=${base_model}
--warmup_ratio=0.05
--fsdp="full_shard auto_wrap"
--fsdp_config="train/fsdp_config_qwen.json"
--bf16=True
--eval_strategy="no"
--logging_steps=1
--save_strategy="no"
--lr_scheduler_type="cosine"
--learning_rate=${lr}
--weight_decay=${weight_decay}
--adam_beta1=0.9
--adam_beta2=0.95
--output_dir="ckpts_1.1/s1-${uid}"
--push_to_hub=${push_to_hub}
--save_only_model=True
--gradient_checkpointing=True
# --accelerator_config='{"gradient_accumulation_kwargs": {"sync_each_batch": true}}'

You’re a lifesaver!, I will try this.

[jnward]

@LiZhangMing To be clear, I got the same results as you-- 0.3667 on AIME (not 0.500 like the authors)

[LiZhangMing]

Hm, I just did the same thing with 8xH200 (using gradient accumulation instead of increasing microbatch size) and got exactly the same result on AIME. I've had better results with s1k-1.1 but was hoping to use the original dataset since it's much faster to train (and I'm running a few experiments)!

Not sure if there are better hyperparams to use? Training loss looks kinda overfit

To be clear, I got the same results as you-- 0.3667 on AIME (not 0.500 like the authors) . I just did the same thing with 8xH200 (using gradient accumulation instead of increasing microbatch size) and got exactly the same result on AIME.

Do these two bolded sentences contradict each other?
Is the first sentence referring to your reproduction results using the s1k dataset with the block size set to 20k?
Is the second sentence referring to your reproduction of the author’s results using s1.1k?

[jnward]

When I said I got exactly the same result, I meant exactly the same as your bad result :)

[jnward]

I'm doing a full run of s1k-1.1 now and will report back

[jnward]

Ok yeah, I ended up with 0.5667 on AIME when training on s1k-1.1, which is what the authors reported. Performance on MATH500 is still a bit worse (I get 0.926 vs authors' 0.944)