Skip to content

Vl dataloading + training improvements - #10

Merged
alay2shah merged 21 commits into
mainfrom
vl-dataloaders
Mar 6, 2026
Merged

Vl dataloading + training improvements#10
alay2shah merged 21 commits into
mainfrom
vl-dataloaders

Conversation

@alay2shah

@alay2shah alay2shah commented Feb 17, 2026

Copy link
Copy Markdown
Collaborator

Collate fn rework:

  • Assistant-only label masking: Replaced TRL's default labels[pad] = -100 with label builder that masks everything except assistant content + <|im_end|> to match the internal stack (old collate had masking/padding issues)
  • Per-sample error handling: Bad images skip the sample with a warning instead of crashing the batch
  • Ray dataset integration: Switched from pre-materialized datasets to ray.train.get_dataset_shard() + ray_dataset_to_hf() consistent w/ prev. PR
  • VLM processor params: max_image_tokens and do_image_splitting now flow from config through to load_vlm_model()

Processor configuration

  • load_vlm_model() now accepts max_image_tokens and do_image_splitting
  • Forces BICUBIC resampling (lost sometimes when starting from a checkpoint), padding config, cache disabled
  • Replaced hardcoded max_image_tokens=256 with configurable parameter

PIL Image loading

  • .convert("RGB") on all paths (handles RGBA, grayscale, palette)
  • PIL safety settings (MAX_IMAGE_PIXELS, LOAD_TRUNCATED_IMAGES)
  • is_image_loadable() to prevent breaks -- again everything to match internal

Distrubuted Ray data filtering for images (much faster)

  • New is_valid_vlm_sft() row filter that validates message structure + actually tries loading each image
  • validate_vlm_sft_format() now fails on unloadable images instead of just warning
  • normalize_columns now covers vlm_sft type
image image

alay2shah and others added 14 commits February 13, 2026 21:36
…le()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…date

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ast, max_image_tokens

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…config safety

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread src/leap_finetune/training_loops/vlm_sft_run.py
@alay2shah
alay2shah requested a review from Rouzbehat78 March 6, 2026 18:12
eval_dataset=test_dataset,
eval_dataset=eval_dataset,
data_collator=collate_fn,
processing_class=processor,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing the processor to the VLMTrainer to save the tokenizer, processor json files on checkpoint saving

…checkpoints

Co-authored-by: Rouzbehat78 <Rouzbehat78@users.noreply.github.com>
@alay2shah
alay2shah merged commit a163a88 into main Mar 6, 2026
1 check passed
@alay2shah
alay2shah deleted the vl-dataloaders branch March 6, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants