[Oneshot] Add validation for empty dataset and enhance oneshot function parameters #1957
Conversation
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
Summary of ChangesHello @ArkaSanka, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the oneshot function by aligning its parameters with the underlying argument dataclasses, which makes the API more complete and intuitive. It also introduces valuable validation for empty datasets and incompatible parameter combinations, such as sequential_targets with an independent pipeline. The tests have been updated accordingly to cover these new parameters and handle potential data-related issues more gracefully. My review focuses on a potential issue in the test suite where preprocessing_func is wrapped in a tuple, which seems to contradict its type hint and the stated goal of the pull request. I have also included a minor style suggestion to improve code maintainability.
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
from
83ebf56 to
b882a49
Compare
ArkaSanka
changed the title
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
from
b882a49 to
845aac4
Compare
|
Hi @kylesayrs, @dsikka, let me know if there are any additional/missing changes to be made. |
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
from
845aac4 to
3d96b12
Compare
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
from
3d96b12 to
3b25767
Compare
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
3 times, most recently
from
2737670 to
7a809b6
Compare
Signed-off-by: Arka Sanka <[email protected]> Refactor oneshot function parameters to use Optional types and enhance documentation Signed-off-by: Arka Sanka <[email protected]>
ArkaSanka
force-pushed
the
oneshot-dataset-params
branch
from
7a809b6 to
59312d5
Compare
| logger.info(f"Dataset type: {type(one_shot_args.get('dataset'))}") | ||
| if isinstance(one_shot_args.get("dataset"), str): | ||
| logger.info(f"Dataset name: {one_shot_args.get('dataset')}") | ||
| logger.info(f"Dataset config: {one_shot_args.get('dataset_config_name')}") | ||
| try: | ||
| # Call oneshot with all parameters as flat arguments | ||
| oneshot( | ||
| **one_shot_args, | ||
| output_dir=tmp_path, | ||
| num_calibration_samples=10, | ||
| pad_to_max_length=False, | ||
| ) | ||
|
|
||
| except ValueError as e: | ||
| if "num_samples should be a positive integer value" in str( | ||
| e | ||
| ) or "Dataset is empty. Cannot create a calibration dataloader" in str(e): | ||
| logger.warning(f"Dataset is empty: {one_shot_args.get('dataset')}") | ||
| pytest.skip(f"Dataset is empty: {one_shot_args.get('dataset')}") | ||
| else: | ||
| raise # Re-raise other ValueError exceptions |
There was a problem hiding this comment.
can you explain why you needed to add these changes? if you are asserting that a certain pathway raises an error in a test, you can do that with
with pytest.raises(ValueError):there are examples of this in the code
3 participants
Fix argument handling in oneshot function #1850
Issue Description
The
oneshotfunction signature inoneshot.pywas missing several parameters that exist in the underlying dataclasses (DatasetArguments,ModelArguments,RecipeArguments). This caused issues when users tried to use these parameters directly, particularly with:sequential_targets: Conflicts occurred between recipe modifiers and direct parameterspreprocessing_func: Returns an error when the dataset is emptypipeline: Not properly validated againstsequential_targetsChanges Made
Parameter Alignment:
oneshotfunction signature to include all missing parameters from the argument dataclassespreprocessing_func,data_collator,raw_kwargs,max_train_samples,pipeline,tracing_ignore,sequential_targetsValidation Logic:
sequential_targetsbetween recipe modifiers and direct parameterspipelinesettings withsequential_targetsTest Improvements:
test_api_inputs.pyto handle all parameters correctlyImpact
These changes ensure that all parameters defined in the argument dataclasses can be used directly with the
oneshotfunction without unexpected behavior. Users can now pass parameters likesequential_targetsandpreprocessing_funcdirectly tooneshotwithout running into cryptic errors or unexpected behavior. The API is now more consistent with its underlying implementation, making it more intuitive to use.