Skip to content

Conversation

aw632
Copy link

@aw632 aw632 commented Mar 25, 2025

When I was following this guide, I noticed that these commands gave different outputs:

python3 /sgl-workspace/sglang/python/sglang/bench_one_batch.py --correct --model meta-llama/Llama-3.2-3B-Instruct
python3 /sgl-workspace/sglang/scripts/playground/reference_hf.py --model-path meta-llama/Llama-3.2-3B-Instruct --model-type text

It turns out that reference_hf.py was autocasting to float16 silently, which was causing this issue. I added this comment in case others come across the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant