Feature/vlm finetune by Rouzbehat78 · Pull Request #13 · Liquid4All/liquid-finetune

Rouzbehat78 · 2026-03-05T19:08:34Z

This pull request introduces several enhancements and improvements to the leap-finetune training pipeline, with a focus on evaluation benchmarks, dataset loading robustness, and documentation clarity. The most significant changes are the addition of automatic evaluation benchmarks during training, improved support for loading datasets from directories of Parquet files, and expanded documentation for both users and developers.
Bug FIX
Bug fixed for per trainer batch size was automatically initalized by HF Trainer self._trainer_batch_size to number of GPUs => fixed now reads the batchsize from configs and global/local steps calculation is being done correclty.

Evaluation Benchmarks Integration

Added a comprehensive evaluation benchmark system, including new files: README.md, base.py, callback.py, and an updated __init__.py, enabling automatic benchmark evaluation during training with distributed all-reduce and wandb logging. [1] [2] [3] [4]
Benchmarks are configured via YAML, support multiple metrics (e.g., short_answer, logprob_zero_shot, mcq_gen and etc), and are fully integrated with HuggingFace Trainer callbacks.

** Resumption of Training **

Added support for reusming the training from where it stopped and resuming from any checkpoint path or latest, while preserving the wandb run ID and optimizer states and etc.

Dataset Loading Robustness

Improved dataset loading logic to support directories containing multiple Parquet files, both in the main loader and in schema validation utilities. This ensures seamless handling of sharded datasets for both training and evaluation. [1] [2] [3]
Enhanced schema validation and sample loading to auto-detect and correctly process directories of Parquet shards.

Documentation and Usability

extended the README and created a README under evaluation/ dir for instructions on how to add custom dataset, metric for the Eval hook.

…nting to multiple parquets

…o a python parsed list. Validate_loader will calidate the schema after preporcessing on sample rows, differnet pre-processsing can be added and functions are callable from the config.yaml

…slurm script

…s to run full eval on various benchmarks and log into wand to track progress and catastrophic forgetting

…o accelatrot dataloader

…mat/correctness of parsing of predicitons

…ame functionality to convert and parse json strings and align with expected HF schema

…to run full eval/benchmark on the eval dataset with their metrics

…t or latest that resolves to last run latest checkppoint

…ext only val datasets

…ashed instead of succesfful/completed

…ldnt be empty at end of train epoch

…tc. So tests have to adhere ot the changes

…nchmarks Move count += 1 inside try block so failed samples are excluded from the denominator. Change hardcoded "cuda" to device.type in VLM autocast to match LLM benchmarks.

Use self.args.per_device_train_batch_size instead of self._train_batch_size which HF Trainer auto-multiplies by world_size, causing double-sharding with Ray.

Eliminates 4 identical evaluate() loops across benchmark subclasses. Subclasses now only implement load_samples() and score_sample().

…r on missing metric Use **kwargs consistently for both logprob and generation paths. Replace vestigial "type" with "metric" in _FACTORY_KEYS. Raise ValueError on missing or unknown metric instead of silently skipping. Remove redundant image_root double-handling in VLM factory.

… logger Remove file-level docstrings from all evaluation module files. Convert logger.info(f"...") to %s style in training loops and trainer_mixins. Replace print() with logger.warning() in config_parser. Fix section header style to use === convention.

Test eval data loaders (format detection, JSONL/JSON loading, limit, normalization, legacy format conversion). Test benchmark base class (failure exclusion, sample caching). Test benchmark config passthrough in YAML parsing.

alay2shah

lgtm

Rouzbehat78 changed the title ~~Feature/vlm finetune~~ [WIP] Feature/vlm finetune Mar 5, 2026

Rouzbehat78 force-pushed the feature/vlm-finetune branch from 4c54c4b to 558446f Compare March 9, 2026 15:28

Rouzbehat78 changed the title ~~[WIP] Feature/vlm finetune~~ Feature/vlm finetune Mar 12, 2026

Rouzbehat78 requested a review from alay2shah March 12, 2026 19:46

Rouzbehat78 force-pushed the feature/vlm-finetune branch from 0d29037 to b94aa98 Compare March 23, 2026 18:16

Rouzbehat78 added 23 commits March 23, 2026 22:42

read the shareded parquets (bucketting) if dataset is a directory poi…

78509d7

…nting to multiple parquets

preprocessing function added to convert json strings in the dataset t…

7b7ff01

…o a python parsed list. Validate_loader will calidate the schema after preporcessing on sample rows, differnet pre-processsing can be added and functions are callable from the config.yaml

adding directives like account, and setup_commands that prepended in …

cc44443

…slurm script

Adding eval/benchamrk callback function in SFTTrainer every eval_step…

df48214

…s to run full eval on various benchmarks and log into wand to track progress and catastrophic forgetting

Fixing double sharding of data by wrapping the Trainer class with a n…

90add50

…o accelatrot dataloader

logging the 20 samples per eval to visualize the tokenizer/prompt for…

2156b0b

…mat/correctness of parsing of predicitons

becnhamrk hook logger cleanup

27381eb

Removing pre-processor and it's logic as normalization achieves the s…

8e04dff

…ame functionality to convert and parse json strings and align with expected HF schema

fixing ruff formats

27be521

format

451d1c1

Add weight decay parsing, log all three LR on wandb

94b05b5

Add Resume training support

7772871

Added a automated Eval callback in the trainer after each eval_steps …

00f9a10

…to run full eval/benchmark on the eval dataset with their metrics

fixing the wandb reusme runid

d5e61ad

Fixing the output direcotries, and resumption form specific checkpoin…

32f1547

…t or latest that resolves to last run latest checkppoint

fixing eval dataset loader to support legacy jsonl formats

d8e83b6

cleanup

756de33

formatting

afc4430

Add and tested the Eval callback for the LLM SFT/DPO traioning with t…

0f4bcf5

…ext only val datasets

close the wandb run once trianing finishes so it doesnt mark it as Cr…

04b6224

…ashed instead of succesfful/completed

removed hte self.metrics.clear as it will get updated, and report wou…

c991ea4

…ldnt be empty at end of train epoch

the output dir structure has changed to make it easier for resuming e…

c756c76

…tc. So tests have to adhere ot the changes

lint

e4f9902

Rouzbehat78 force-pushed the feature/vlm-finetune branch from 32c566d to e4f9902 Compare March 23, 2026 23:24

normalize the log probs per token lenght

7b65bdc

Rouzbehat78 and others added 8 commits March 24, 2026 00:16

linting!

8a2dbcc

Fix failed sample count deflating scores and hardcoded cuda in VLM be…

64aa1f7

…nchmarks Move count += 1 inside try block so failed samples are excluded from the denominator. Change hardcoded "cuda" to device.type in VLM autocast to match LLM benchmarks.

Fix double batch size multiplication in RayDataLoaderMixin

b34cde4

Use self.args.per_device_train_batch_size instead of self._train_batch_size which HF Trainer auto-multiplies by world_size, causing double-sharding with Ray.

Move evaluate() to Benchmark base class, make score_sample abstract

45e1e9b

Eliminates 4 identical evaluate() loops across benchmark subclasses. Subclasses now only implement load_samples() and score_sample().

Fix ruff lint and format issues

7dab40f

alay2shah approved these changes Mar 24, 2026

View reviewed changes

Rouzbehat78 merged commit cb06430 into main Mar 24, 2026
1 check passed

Rouzbehat78 deleted the feature/vlm-finetune branch March 24, 2026 16:39

Navigation Menu

Uh oh!

Feature/vlm finetune - #13

Feature/vlm finetune#13
Rouzbehat78 merged 32 commits into
mainfrom
feature/vlm-finetune

Rouzbehat78 commented Mar 5, 2026 •

edited

Loading

Uh oh!

alay2shah left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Rouzbehat78 commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alay2shah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Rouzbehat78 commented Mar 5, 2026 •

edited

Loading