Distinguishing model failure vs task inadmissibility in evaluation

[finkeissen]

In some evaluation workflows, a task can become ill-posed rather than simply failed.

For example:

• missing or contradictory context,
• ambiguous labeling criteria,
• evaluation signals that cannot be interpreted meaningfully.

Do you distinguish between model failure and task inadmissibility during evaluation?

I’m curious whether this is tracked explicitly in Kiln workflows or treated as a regular failure.

[aniruddhaadak80]

I would treat those as separate outcomes. A model failure says the system had a well-posed task and produced the wrong or low-quality result. Task inadmissibility says the evaluation target itself was not coherent enough to judge meaningfully.nnIf both end up in the same bucket, the signal gets noisy very quickly. You lose the ability to distinguish model weakness from dataset or rubric weakness, which matters a lot when people start making tuning decisions from the evaluation results.

[finkeissen]

Great point, @aniruddhaadak80. I’m actually working on similar evaluation patterns over at my profile (github.com), and separating 'model failure' from 'task inadmissibility' has been a game-changer for debugging prompts.
In Kiln, having an explicit 'Inadmissible' label would prevent these cases from unfairly dragging down model performance metrics. Is this something that could be added to the Kiln workflow or schema?