Skip to content

Add validation for prompt name uniqueness across the service. #833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 6, 2025

Conversation

yasmewad
Copy link
Contributor

@yasmewad yasmewad commented Jul 31, 2025

Add validation for prompt name uniqueness across services

  • Implements validation to prevent prompt naming conflicts within Smithy services. Prompts are now validated for case-insensitive uniqueness across both service-level and operation-level definitions.

Changes

  • Added a validator for model level uniqueness when the prompt is used to have unique names.
  • Added a check in MCP Server to have validation when prompts are loaded for normalized name to be unique.

The prompts are checked for uniqueness of name case-insensitively following same rules as defined for ShapeIds in the semantics of a Smithy Model.

While shape ID references within the semantic model are case-sensitive, no two shapes in the semantic model can have the same case-insensitive shape ID. This restriction makes it easier to use Smithy models for code generation in programming languages that do not support case-sensitive identifiers or that perform some kind of normalization on generated identifiers (for example, a Python code generator might convert all member names to lower snake case). To illustrate, com.Foo#baz and com.foo#BAZ are not allowed in the same semantic model. This restriction also extends to member names: com.foo#Baz$bar and com.foo#Baz$BAR are in conflict.

Examples of conflict

Invalid - would cause validation errors

@prompts({
    get_employee_info: {
        description: "Service-level prompt"
        template: "Get employee info"
    }
})
service EmployeeService {
    operations: [GetEmployeeDetails]
}

@prompts({
    Get_Employee_Info: {  // Same name, different case
        description: "Operation-level prompt"
        template: "Get employee details"
    }
})
operation GetEmployeeDetails {
    // ...
}

Validation Error: Prompt name 'Get_Employee_Info' conflicts with existing prompt 'get_employee_info' in service EmployeeService (case-insensitive comparison)

After (Valid - unique names):

@prompts({
    get_employee_info: {
        description: "Service-level prompt"
        template: "Get employee info"
    }
})
service EmployeeService {
    operations: [GetEmployeeDetails]
}

@prompts({
    get_employee_details_operation: {  // Unique name
        description: "Operation-level prompt"
        template: "Get employee details"
    }
})
operation GetEmployeeDetails {
    // ...
}

Testing

  • SmithyBuild on the invalid model now throws error:

Validation changes

Screenshot 2025-07-30 at 5 23 12 PM

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


/// Definition of the prompt template
value: PromptTemplateDefinition
}

@pattern("^[a-zA-Z0-9]+(?:_[a-zA-Z0-9]+)*$")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently keeping it as characters, allowed with _ separators and numbers.

@yasmewad yasmewad force-pushed the unique-prompt-names branch from 8736e35 to 46202c2 Compare July 31, 2025 02:47
var promptTemplateDefinition = entry.getValue();
var templateString = promptTemplateDefinition.getTemplate();

// Generate the new prompt name with service prefix
var promptName = serviceName + "__" + originalPromptName;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some clients (e.g. q) will automatically prefix tool names with the servers that provide them. Will our manual implementation clash?

Copy link
Contributor Author

@yasmewad yasmewad Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL - that tools get prefixed automatically!

For prompts, that isn't the case.

As seen in this image: its from Q CLI. I was able to invoke the prompt appropriately with this new name.

Before:
Screenshot 2025-07-24 at 4 24 45 PM

After:
Screenshot 2025-07-31 at 9 18 31 AM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to do this for tools in #686, but it turned out to be a bad idea because we started hitting size limit on the tool names and reverted it in #698

Copy link
Contributor Author

@yasmewad yasmewad Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that is a good point. For prompts, I don't see a limit in the MCP schema or host like tools. I can play around and verify it. However, we can choose to enforce a limit ourselves on the prompts and service name. Alternatively expose an id field in the trait that let's the customer define what its called?

We can always reduce this in future if we want. It should be a good start for now?

For now IMO this disambiguation works pretty well across services. For instance, DynamoDB__safe_delete_table and Glue__safe_delete_table.

What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would avoid till we have a use case to add.

Copy link
Contributor Author

@yasmewad yasmewad Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. so as part of the PR, the uniqueness validator and naming @pattern is still good to push? If yes, I will update this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline, and we also wanted a check in terms of server side uniqueness. I have added that check instead of adding the ServiceName___PromptName convention for now. If we see issues with the server in future, we should add the unique naming logic at that time.

@yasmewad yasmewad force-pushed the unique-prompt-names branch 3 times, most recently from 0a4dec5 to 81b0d9a Compare August 6, 2025 02:03

if (prompts.containsKey(normalizedName)) {
var existingPrompt = prompts.get(normalizedName);
throw new RuntimeException(String.format(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a better exception I can throw here that will probably alert user that the server isn't loading due to prompt names?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message should be surfaced to the user.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the only way would be to surface this at client during runtime, when the MCP Client starts it up.

…ll as in a single smithy model at build time.
@yasmewad yasmewad force-pushed the unique-prompt-names branch from 81b0d9a to d550570 Compare August 6, 2025 21:02
@yasmewad yasmewad merged commit fe2c41f into smithy-lang:main Aug 6, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants