fix: Apply guardrails transformations to LLM inputs and bot outputs.

[lapinek]

Description

Currently, when input and output rails process (that is, transform or redact) user and bot messages, the processed versions are not used:

• LLM flows use original $event.final_transcript instead of processed $user_message.
• Bot utterances use the original $text instead of processed $bot_message.
• This allowed sensitive, unfiltered data to leak to LLM and users despite guardrails.

To reproduce, you can run this config against the current develop (requires OPENAI_API_KEY):

#config.yml

colang_version: "2.x"

models:
  - type: main
    engine: openai
    model: gpt-4o-mini

# main

import core
import llm
import guardrails

flow input rails $input_text
    global $user_message
    $user_message = "{$input_text}, Dick"

flow output rails $output_text
    global $bot_message
    $bot_message = "{$output_text}, and Harry"

flow main
  activate llm continuation

poetry run nemoguardrails chat --config /path/to/config
> Echo this: Tom  

Expected output:

Tom, Dick, and Harry

Actual output:

Tom

With patch provided in this PR, the output should be:

Tom, Dick., and Harry

Related Issue(s)

• Could be related to: bug: Cannot sanitize user input #882

Mentions

@schuellc-nvidia, @drazvan - since you’ve contributed most to the affected files, your review would be much appreciated!

Checklist

• I've read the CONTRIBUTING guidelines.
• I've updated the documentation if applicable.
• I've added tests if applicable.
• @mentions of the person or team responsible for reviewing proposed changes.

[lapinek]

In addition to the provided example and test, you can observe the unfixed behavior in the following branches (we’re planning to open PRs for these as well):

• https://github.com/pangeacyber/NeMo-Guardrails/tree/feature/pangea-ai-guard
• https://github.com/pangeacyber/NeMo-Guardrails/tree/feature/sensitive_data_detection-v2-example

[schuellc-nvidia]

Thank you @lapinek, will take a look!
@Pouyanpi Can you also take a look at this!

[github-actions]

Documentation preview

https://nvidia.github.io/NeMo-Guardrails/review/pr-1297

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@0d6fa42). Learn more about missing BASE report.
⚠️ Report is 25 commits behind head on develop.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #1297   +/-   ##
==========================================
  Coverage           ?   70.70%           
==========================================
  Files              ?      161           
  Lines              ?    16312           
  Branches           ?        0           
==========================================
  Hits               ?    11533           
  Misses             ?     4779           
  Partials           ?        0           

Flag	Coverage Δ
python	70.70% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[schuellc-nvidia]

Changes look good to me!

Ideally, we would also have a test that involves the flows generating user intent for unhandled user utterance and continuation on unhandled user utterance that are affected by it.

[lapinek]

Changes look good to me!

Ideally, we would also have a test that involves the flows generating user intent for unhandled user utterance and continuation on unhandled user utterance that are affected by it.

@schuellc-nvidia, thank you for the review!

Would it be OK if we add these tests in a follow-up? I'd like to take a closer look at the dialog flows first to ensure meaningful coverage. I believe the already included test verifies that the rails transformations are applied. We're relying on this behavior in upcoming PRs.

Update:

• feat: add Pangea AI Guard community integration #1300
• docs(examples): add Colang 2.0 example for sensitive data detection #1301

[schuellc-nvidia]

Changes look good to me!
Ideally, we would also have a test that involves the flows generating user intent for unhandled user utterance and continuation on unhandled user utterance that are affected by it.

@schuellc-nvidia, thank you for the review!

Would it be OK if we add these tests in a follow-up? I'd like to take a closer look at the dialog flows first to ensure meaningful coverage. I believe the already included test verifies that the rails transformations are applied. We're relying on this behavior in upcoming PRs.

Update:

• feat: add Pangea AI Guard community integration #1300
• Add Colang v2 example for sensitive data detection #1301

Yes, that's fine with me.

[Pouyanpi]

Thank you @lapinek , this is a critical security fix with solid implementation 👍🏻 It is ready to merge 🚀

Would you please gpg sign your commits following contributing guidelines?

[lapinek]

@Pouyanpi, thanks for your review! I've signed the commits.

The reason I initially used "Signed-of-by..." was that it seemed like a valid alternative according to the contribution guidelines: https://github.com/NVIDIA/NeMo-Guardrails/blob/52ac7edc18e5b6fb1900b639b5c6734dca09b918/CONTRIBUTING.md#summary. Since GPG signature is required, maybe that wording could be more explicit. But I am happy to sign, no problem 🙂