Skip to content

Conversation

@jwpark33
Copy link

@jwpark33 jwpark33 commented Feb 12, 2026

Purpose

Fix --hf-overrides normalization for TorchAO (and general HF overrides) so CLI inputs are handled consistently.

This PR adds normalization logic in EngineArgs to support:

  • JSON object string input, e.g. --hf-overrides '{"quantization_config_file": "..."}'
  • JSON file input via @path, e.g. --hf-overrides '@/path/to/overrides.json'

It also adds tests and docs updates:

  • vllm/engine/arg_utils.py: add _normalize_hf_overrides()
  • tests/engine/test_arg_utils.py: add normalization tests for string/file/invalid cases
  • vllm/config/model.py: clarify accepted string formats in docstring
  • docs/features/quantization/torchao.md: add online TorchAO usage examples with --hf-overrides

Test Plan

Run targeted arg-utils tests for hf-overrides:

pytest -q tests/engine/test_arg_utils.py

Test Result

All passed


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify
Copy link

mergify bot commented Feb 12, 2026

Documentation preview: https://vllm--34430.org.readthedocs.build/en/34430/

@mergify mergify bot added documentation Improvements or additions to documentation bug Something isn't working labels Feb 12, 2026
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a useful enhancement by allowing --hf-overrides to be specified as either a JSON string or a file path. The implementation is clean, and the inclusion of tests and documentation is commendable. I've identified a couple of minor correctness issues where the loaded JSON is not validated to be a dictionary, which could lead to less clear error messages downstream. I've provided suggestions to address these. Overall, this is a solid improvement.

raise ValueError("hf_overrides file path is empty.")
try:
with open(path, encoding="utf-8") as handle:
self.hf_overrides = json.load(handle)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The loaded JSON from the file should be validated to be a dictionary. The hf_overrides argument is expected to be a dictionary of overrides. Currently, if the file contains a JSON primitive (like a string or a number), it will be parsed successfully but will cause an AttributeError later when .items() is called on it. This can be confusing for the user. It's better to fail early with a clear error message.

                    loaded_json = json.load(handle)
                    if not isinstance(loaded_json, dict):
                        raise ValueError(
                            f"hf_overrides file must contain a JSON object: {path}"
                        )
                    self.hf_overrides = loaded_json


if re.match(r"(?s)^\s*{.*}\s*$", raw):
try:
self.hf_overrides = json.loads(raw)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similar to the file loading case, the loaded JSON from the string should be validated to be a dictionary. If the string is a valid JSON primitive (e.g., \"a string\" or 123), it will be parsed successfully but will cause an AttributeError later. It's better to fail early with a clear error message ensuring the provided string is a JSON object.

                loaded_json = json.loads(raw)
                if not isinstance(loaded_json, dict):
                    raise ValueError("hf_overrides string must be a JSON object.")
                self.hf_overrides = loaded_json

@mergify
Copy link

mergify bot commented Feb 12, 2026

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jwpark33.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Feb 12, 2026
@mergify mergify bot removed the needs-rebase label Feb 12, 2026
@jwpark33 jwpark33 marked this pull request as ready for review February 12, 2026 14:54
Signed-off-by: jwpark33 <[email protected]>
@jwpark33 jwpark33 changed the title [Bugfix] FIx torchao-config [Bugfix] FIx TorchAO config bugs Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant