Allow dataset-level prompt template for more flexible benchmark tests

[S3annnyyy]

Description

I’m interested in running custom benchmark tests for my LLM application in a CI pipeline using the existing accuracy_adapter.py. However, I’ve run into an issue: there is no straightforward way to add additional prompt instructions to the dataset input to enforce consistent formatting of model outputs.

Example dataset input

{
    "input": "What does IRR stand for in financial analysis?\nA) Income Recurrent Rate\nB) Internal Rate of Return\nC) Increase Risk Return",
    "target": "B) Internal Rate of Return"
},

GPT-4o output

The correct answer is:

B) Internal Rate of Return ✅

In financial analysis, the Internal Rate of Return (IRR) is the discount rate that makes the net present value (NPV) of all future cash flows from a project or investment equal to zero. It’s commonly used to evaluate the profitability of investments.

This response fails evaluation in accuracy_adapter.py because it contains extra explanation. To fix this, I need to prepend an instruction like:

You are part of a QA team assessing your AI system to ensure it will not generate irrelevant answers from MCQ related questions. Answer the following multiple choice questions where your response should be of the following format: '$LETTER) $FULL_ANSWER' (without quotes) where LETTER is one of the ABCD and FULL_ANSWER is the full answer text that correlates to the letter

Current Workarounds

Method 1: Adding either connector_pre_prompt or connector_post_prompt or system_prompt to the custom connector
Drawback: This ties the connector to a 1-1 relationship with the type of benchmark dataset. If I were to expand my benchark tests beyond MCQ format, I would need to create a new connector with new system prompt which is not ideal. Ideally one connector should be able to run multiple

Method 2: Adding the instruction to every single input
Drawback: Tedious, conflicts with DRY principle

Method 3: Creating a new metric adapter that takes in system prompt
Drawback: Metric adapters evaluate after model response, not before.

What I propose:

Modifying src/domain/services/task_manager.py and adding a prompt_template key to the dataset file so that each prompt can contain additional prompt and input question.

src/domain/services/task_manager.py

def _generate_prompts(self, dataset_entity: DatasetEntity) -> list[PromptEntity]:
        logger.info(self.INFO_GENERATING_PROMPTS)
        prompt_prefix = dataset_entity.prompt_template or ""                         #NEW ADDITION HERE
        return [
            PromptEntity(
                index=index,
                prompt=f"{prompt_prefix}\n\n{example.pop("input", "")}",            #NEW ADDITION HERE
                target=example.pop("target", ""),
                reference_context=example.pop("reference_context", ""),
                model_prediction=None,
                evaluation_result={},
                additional_info={key: value for key, value in example.items()},
            )
            for index, example in enumerate(dataset_entity.examples, 1)

custom-test.json

{
    "name": "Custom Test",
    "description": "Custom Test to test LLM Knowledge",
    "license": "No license",
    "reference": "No reference",
    "prompt_template": "<prompt will be added here>",
    "examples": [
        {
            "input": "What does IRR stand for in financial analysis?\nA) Income Recurrent Rate\nB) Internal Rate of Return\nC) Increase Risk Return",
            "target": "B) Internal Rate of Return"
        },
      ...
    ]
}

Benefits

1. Keeps connectors generic (no need for one connector per dataset format).
2. Minimal, non-breaking changes to the existing codebase.
3. Improves flexibility for developers creating custom benchmarks.

If there exist functionalities that I’ve missed which already solve this problem, I’d be happy to learn about them. Otherwise, I’m also happy to contribute a PR for this improvement as I have tested it on my end with success

[aivf-kaydenyip]

Hi, thanks for the suggestion! We acknowledge the current limitation of not able to specify prompt template at Test or Dataset level and the mentioned workarounds are not the ideal experience for users. We have included your feature request into our roadmap and will take in your suggestion in our design considerations. You can expect this feature to be available in our next major feature release.

[S3annnyyy]

@aivf-kaydenyip thanks for the update!