[Speculative Decoding] Add speculators config support

[dsikka]

Purpose

• Support Speculators Config
• Adds support to load models saved / converted using the Speculators repository
• Adds support to serve Eagle3 Llama models trained + saved using the Speculators format

Summary of Changes

• Introduces a SpeculatorsConfig to load models saved with the speculators format
• Updates the llama_eagle3 definition to optionally add norm_before_residual - a field saved by eagle3 speculators
• Allows loading / serving of the speculator without the requirement of providing a speculative_config as input. In order to achieve this functionality, we optionally update the model based on if the runner is a draft or not, and fill the details of the speculative_config based on the config

Speculators Config:

• Example config can be found here: https://huggingface.co/nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717/blob/main/config.json

API

• You can load / run speculators through the LLM Engine as well as through vllm serve
• You only need to provide a model path to your speculator - the target model will be read from the config
• You can still load models through the speculative_config pathway - this allows you to override any arguments in your config (such as num_speculative_tokens)
• Note: you cannot currently override the target model - it will be read from the config

E.g

VLLM_USE_V1=1 vllm serve "nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717"

• Override the number of speculative tokens from 3 to 5

VLLM_USE_V1=1 vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic --speculative_config '{"model":"nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717", "num_speculative_tokens": 5, "method": "eagle3"}'

Test Plan

• Tested target models

1. RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic
2. meta-llama/Meta-Llama-3.1-8B-Instruct

Test Result - GuideLLM Benchmarking:

vLLM:

VLLM_USE_V1=1 vllm serve nm-testing/SpeculatorLlama3-1-8B-Eagle3-converted-0717 --port 7600 >output_speculators_llama.tx

Per-position acceptance rate (dense target)

[0.812 0.618 0.446]
conditional
[0.812 0.761 0.722]

FP8 Quantized Target:

[0.8   0.604 0.436]
conditional
[0.8   0.755 0.723]

Follow-up

• Qwen support
• Eagle support

[gemini-code-assist]

Code Review

This pull request introduces support for Speculators Config. There are a few critical issues that need to be addressed:

1. There's a consistent misuse of self.config_dict instead of self.config in vllm/transformers_utils/configs/speculators/base.py, which will cause runtime errors.
2. A typo in a method name update_defualts in vllm/transformers_utils/configs/speculators/eagle.py will prevent it from being called, breaking the configuration logic for Eagle-1 models.

[robertgshaw2-redhat]

👀

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[aarnphm]

First round

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dsikka.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[aarnphm]

One tiny comment, otherwise LGTM

[mgoin]

Nice work!