Sampling is the process of choosing the next token from the model's predicted probabilities.
The model does not simply return one guaranteed answer. It produces scores for many possible next tokens. Sampling settings decide how the final token is selected.
Greedy decoding always chooses the most likely next token. This is predictable, but it can be repetitive or less creative.
Temperature controls randomness.
Lower temperature makes output more focused and predictable. Higher temperature makes output more varied and risky.
For structured extraction, use low temperature. For brainstorming, a higher temperature may be acceptable.
Top-p limits choices to a group of likely tokens whose total probability reaches a threshold. This can reduce strange low-probability choices while still allowing variety.
Sampling affects reliability. A model used for JSON extraction, routing, or policy decisions should not behave wildly. A model used for creative writing can tolerate more variation.
Sampling also affects testing. If outputs vary, tests must check properties rather than exact text.
- Sampling chooses the next token.
- Temperature controls randomness.
- Top-p limits the candidate set.
- Production systems should choose settings based on task risk.
- Use low randomness for factual or structured tasks.
- Validate generated JSON or code.
- Keep sampling settings visible in configuration.
- Test prompts under the settings used in production.