Skip to content

Tool call testing#19

Merged
Rouzbehat78 merged 15 commits intomainfrom
tool-call-testing
Apr 21, 2026
Merged

Tool call testing#19
Rouzbehat78 merged 15 commits intomainfrom
tool-call-testing

Conversation

@alay2shah
Copy link
Copy Markdown
Collaborator

This PR adds tool call format guardrails to the training pipeline.

  • Detects LFM2, LFM2.5, structured (OpenAI), and foreign formats from a sample of the data
  • Auto-strips <|tool_response_start|> from role=tool messages to prevent double wrapping (the LFM2 template adds these itself)
  • Auto-strips <|tool_list_start|> when training LFM2.5 models
  • Auto-converts OpenAI-style tool_calls fields to pre-baked bracket notation
  • Hard errors on foreign formats (<tool_call> XML etc.) with a message telling you exactly what to do
  • Warns on format mismatches it can't auto-fix (e.g. missing <|tool_list_start|> for LFM2)

Pipeline integration is light. The normalization runs as a Ray .map() step between column normalization and filtering — same pattern as existing transforms. Existing files got ~10 lines each. model_name flows from the YAML config through DatasetLoader so the pipeline knows which model family to validate against.

Also re-enables preprocess_fn — users can now specify preprocess_fn: "my_module.my_func" in their YAML config for custom dataset transforms.

@alay2shah alay2shah requested a review from Paulescu March 30, 2026 16:33
@alay2shah alay2shah requested a review from Rouzbehat78 April 20, 2026 14:13
…ut an invalid stinrg that isnt parsable as they include special characters that need escaping with \
…with correct escaping character \, passes all the unit tests now
Copy link
Copy Markdown
Contributor

@Rouzbehat78 Rouzbehat78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did add few edge cases in the pytests where the tool_calls_to_pythonic converter was outputting invalid strings that arent parsable when a special character existed in the string like (", , \n, etc)
I also pushed the small fix, so we use json dumps for those cases that will make sure it's parsable stings.

cache_dataset: bool = False
# Optional preprocessing function: takes Ray Dataset, returns Ray Dataset
# Applied before validation - use for custom filtering, transforms, joins, etc.
preprocess_fn: Callable | None = field(default=None, repr=False)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we using this anywhere?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agents sometime use it but it's relatively harmless alone

@alay2shah
Copy link
Copy Markdown
Collaborator Author

Lgtm, just approve

@Rouzbehat78 Rouzbehat78 self-requested a review April 21, 2026 17:57
@Rouzbehat78 Rouzbehat78 merged commit d017458 into main Apr 21, 2026
1 check passed
@Rouzbehat78 Rouzbehat78 deleted the tool-call-testing branch April 21, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants