Skip to content

Conversation

derekk-nm
Copy link
Contributor

SUMMARY:

  • updates to Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 server configuration to attempt to get the accuracy workflow to run.
  • RedHatAI/DeepSeek-R1-0528-quantized.w4a16 was missing its base model, and a server option.
  • HuggingFaceTB/SmolLM3-3B is updated with an actual ground-truth gsm8k value.

TEST PLAN:
models were passed through the accuracy workflow.
the Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 model is just too big to test w/out access to a larger system.

@tmuttaki
Copy link

tmuttaki commented Sep 4, 2025

I saw these models in @debroy-rh 's small list of models for code coverage but these are not present in model validation config. Are you going to add these as well?

  • deepseek-ai/DeepSeek-V2-Lite
  • microsoft/Phi-3-medium-4k-instruct
  • Qwen/Qwen2-57B-A14B-Instruct
  • RedHatAI/Meta-Llama-3-8B-Instruct-FP8-KV
  • RedHatAI/Mixtral-8x7B-Instruct-v0.1-FP8
  • TechxGenus/Meta-Llama-3-8B-Instruct-GPTQ

Copy link

@debroy-rh debroy-rh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look good, thanks Derek.

@derekk-nm
Copy link
Contributor Author

Thanks for the observation @tmuttaki .
Those models appear in the neuralmagic/lm-eval-configs/models directory because they were part of testing prior to the NeuralMagic acquisition by Red Hat. They haven't been included as part of the Third Party Model Validation program initiated at the beginning of Red Hat Summit. (this is the source I'm using for models in that program).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants