AAP-54071: Fix raw tool_call in the message responses

[de1987]

Jira Issue: AAP-54071

Description

As reported, there were some prompts that the responses contains raw tool_call.
By adjusting the system prompt instructions the issue was fixed for Granite models.

Testing

Steps to test

1. Pull down the PR
2. make build
3. make run
4. Open the chatbot UI and ask for "How do I upgrade to latest version of Ansible Automation Platform?"

Scenarios tested

Verified with at least 20 prompts asking questions about AAP, create Ansible playbooks...

Production deployment

• This code change is ready for production on its own
• This code change requires the following considerations before going to production:

[mabashian]

I'm not an expert in the chatbot but this approach seems reasonable to me

[romartin]

Hey @de1987 ,

thanks, looking good, although let me share a few comments:

• I use to wrap code blocks, as commented below, and preferable use of MarkDown syntax in general.

• Which models have you tested? We need to be sure that it works with Gemini and Granite, if possible, and we don't care about OpenAI at this point

• The files you changed, in fact, are not being used on the deployments in our clusters. Those files you changed are only applicable when running locally the chatbot. So it is fine doing this change, but if we want this change applied to our deployed chatbots, you'll have to also update our -ops repo, such as this PR does

• As a final comment, we have an evaluation tool for the chatbot, that evaluates queries around AAP. I tihnk it should be good running it, once changing the sys. prompt, just to be sure we're not breaking anything

[romartin]

I think it's better to wrap those kind of JSON blocks by using code block (an example here), so the LLM knows exactly when the block starts and end.

[de1987]

I'm using Granite. I addressed all your suggestions and run the evaluation tool:

...
Processing query 84: What is subscription usage?...
✅ Success for query 84

[de1987]

@romartin PR to deploy in our clusters https://github.com/ansible/ansible-wisdom-ops/pull/1596

[TamiTakamiya]

Although I am not sure how much efforts we want to put on Granite LLM specific changes, I think the changes are clear and looks good to me.