Distinguishing model failure vs task inadmissibility in evaluation

[finkeissen]

In some evaluation workflows, a task can become ill-posed rather than simply failed.

For example:

• missing or contradictory context,
• ambiguous labeling criteria,
• evaluation signals that cannot be interpreted meaningfully.

Do you distinguish between model failure and task inadmissibility during evaluation?

I’m curious whether this is tracked explicitly in Kiln workflows or treated as a regular failure.