training context length

[ggg-s]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

One question: I notice that most models are trained with a context length of around 2K. When dealing with ultra-long contexts—such as 16K, 32K, or even 128K—can the draft model still maintain reasonable accuracy?

Reproduction

.

Environment

.

[jiapingW]

Rope can perform length extrapolation, but there will likely be a performance loss when generalizing from 2k to 128k, since the data distribution of 128k is already inconsistent with that of 2k.