[SIT] EISW-204381: Add per output threshold by svigh · Pull Request #34352 · openvinotoolkit/openvino

svigh · 2026-02-26T10:11:22Z

Details:

Changed argument parsing to be able to pass different threshold value per layer
Moved mAP metric helpers to separate files
Change default value for NRMSE metric to 0.02 instead of

Tickets:

EISW-204381

AI Assistance:

AI assistance used: yes
Generated the per layer value parsing of argument strings, tested manually on PTL RVP

…ce in separate parsing files

Copilot

Pull request overview

This PR adds support for per-output (per-layer) threshold specification across all metric comparison modes in the single-image-test tool. Previously, all metrics used global threshold values; now users can specify different thresholds for different output layers using the syntax "layer1:value1;layer2:value2" or continue using single global values.

Changes:

Extracted mAP metric helper functions to separate files (map_metric_helpers.hpp/cpp) for better code organization
Implemented per-layer threshold parsing infrastructure in new argument_parse_helpers.hpp/cpp files
Changed NRMSE default threshold from 1.0 to 0.02
Updated all metric test functions to support per-layer thresholds while maintaining backward compatibility with single global values

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
map_metric_helpers.hpp	Declares structures (Detection, MatchResult) and functions for mAP calculation, extracted from main.cpp
map_metric_helpers.cpp	Implements IoU calculation, detection parsing, matching, and average precision computation for mAP metric
argument_parse_helpers.hpp	Declares all command-line flags (previously in main.cpp), provides utility classes (CaselessEq), and declares per-layer value parsing functions
argument_parse_helpers.cpp	Defines all DEFINE_string/DEFINE_* flags with updated help text mentioning per-layer support, implements parsePerLayerValues and getValueForLayer functions, and moves parseCommandLine function from main.cpp
main.cpp	Integrates per-layer threshold parsing into all metric test functions (RAW, CoSim, RRMSE, NRMSE, L2Norm, PSNR, Classification, SSD, YOLO, MeanIoU), removes flag definitions and helper code now in argument_parse_helpers, and updates PSNR to handle multi-output models differently

src/plugins/intel_npu/tools/single-image-test/map_metric_helpers.cpp

Copilot · 2026-02-26T13:59:41Z

src/plugins/intel_npu/tools/single-image-test/main.cpp

+    // If there's only one output, use the global PSNR metric
+    if (outputs.size() == 1) {
+        auto refOutput = npu::utils::parseTensorsAsFP32(references);
+        auto actOutput = npu::utils::parseTensorsAsFP32(outputs);

-    auto result = utils::runPSNRMetric(actOutput, refOutput, dstHeight, dstWidth, scaleBorder, normalizedImage);
+        auto result = utils::runPSNRMetric(actOutput, refOutput, dstHeight, dstWidth, scaleBorder, normalizedImage);

-    if (FLAGS_psnr_reference - result > FLAGS_psnr_tolerance) {
-        std::cout << "Absolute difference between actual value " << result << " and reference value "
-                  << FLAGS_psnr_reference << " larger then tolerance " << FLAGS_psnr_tolerance << std::endl;
-        return false;
+        // Get per-layer or global values
+        std::string layerName = outputs.begin()->first;
+        double psnrReference = utils::getValueForLayer(referenceMap, layerName);
+        double psnrTolerance = utils::getValueForLayer(toleranceMap, layerName);
+
+        if (psnrReference - result > psnrTolerance) {
+            std::cout << "Absolute difference between actual value " << result << " and reference value "
+                      << psnrReference << " larger then tolerance " << psnrTolerance << std::endl;
+            return false;
+        }
+    } else {
+        // For multiple outputs, compute PSNR per layer
+        for (const auto& [layerName, outputTensor] : outputs) {
+            auto refIterator = references.find(layerName);
+            OPENVINO_ASSERT(refIterator != references.end());
+
+            TensorMap singleOutput = {{layerName, outputTensor}};
+            TensorMap singleReference = {{layerName, refIterator->second}};
+
+            auto refOutput = npu::utils::parseTensorsAsFP32(singleReference);
+            auto actOutput = npu::utils::parseTensorsAsFP32(singleOutput);
+
+            auto result = utils::runPSNRMetric(actOutput, refOutput, dstHeight, dstWidth, scaleBorder, normalizedImage);
+
+            // Get per-layer values
+            double psnrReference = utils::getValueForLayer(referenceMap, layerName);
+            double psnrTolerance = utils::getValueForLayer(toleranceMap, layerName);
+
+            std::cout << "Layer: " << layerName << " - ";
+            if (psnrReference - result > psnrTolerance) {
+                std::cout << "Absolute difference between actual value " << result << " and reference value "
+                          << psnrReference << " larger then tolerance " << psnrTolerance << std::endl;
+                allPassed = false;
+            } else {
+                std::cout << "PASSED (PSNR: " << result << ", Reference: " << psnrReference << ")" << std::endl;
+            }
+        }
    }


The PSNR test behavior has changed in a potentially breaking way. Previously, for models with multiple outputs, all outputs were combined into a single PSNR calculation. Now, the code branches at line 1853: if there's exactly one output, it combines all tensors (which is just one), but if there are multiple outputs, it evaluates each layer separately at lines 1870-1895.

This is a functional change, not just adding per-layer threshold support. If this is intentional, it should be documented in the PR description. If not, the multi-output case should still combine all outputs like the original implementation did, while supporting per-layer thresholds for the combined result.

Copilot · 2026-02-26T13:59:41Z

src/plugins/intel_npu/tools/single-image-test/argument_parse_helpers.cpp

+    } catch (...) {
+        // Not a single number, parse as key:value pairs


The catch-all exception handler at line 220 silently treats any invalid single-value input (e.g., "abc") as a key:value format. If the user mistakenly provides an invalid single value, it will be silently ignored and the default will be used without any warning. Consider adding a warning message inside the catch block to inform users that the single-value parse failed and it will be treated as key:value format, or validate that at least one colon exists before proceeding to key:value parsing.

Suggested change

} catch (...) {

// Not a single number, parse as key:value pairs

} catch (const std::exception& e) {

// Not a single number. If there is no colon, we cannot interpret this as "layer:value".

if (str.find(':') == std::string::npos) {

std::cerr << "Warning: Failed to parse per-layer value '" << str

<< "' as a single number and no 'layer:value' pairs were found. "

<< "Using default value " << defaultValue << "." << std::endl;

return result;

}

// Otherwise, fall through and try to parse as key:value pairs.

…rs.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

svigh added 3 commits February 23, 2026 15:29

Add per layer parsing of thresholds

3a11287

Add argument parser helper files

fbdfa4c

Move map metric helpers to separate files, add argument parse namespa…

bd195b0

…ce in separate parsing files

svigh requested review from a team as code owners February 26, 2026 10:11

github-actions bot added the category: NPU OpenVINO NPU plugin label Feb 26, 2026

svigh requested a review from Copilot February 26, 2026 13:52

Copilot started reviewing on behalf of svigh February 26, 2026 13:52 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

svigh and others added 7 commits February 27, 2026 10:46

Update src/plugins/intel_npu/tools/single-image-test/map_metric_helpe…

4002c07

…rs.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/plugins/intel_npu/tools/single-image-test/map_metric_helpe…

5379423

…rs.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Change psnr back to single threshold value

d87c7a7

Merge branch 'master' into EISW-204381_add_per_output_threshold

e6f7cce

Merge branch 'master' into EISW-204381_add_per_output_threshold

bcba745

Merge branch 'master' into EISW-204381_add_per_output_threshold

53f034d

Remove unused exception variable

13d857f

fcmiron approved these changes Mar 16, 2026

View reviewed changes

svigh added this pull request to the merge queue Mar 16, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 16, 2026

svigh added 2 commits March 16, 2026 15:52

Merge branch 'master' into EISW-204381_add_per_output_threshold

fb614e4

Fix copyright headers

5ee3929

github-actions bot added the category: build OpenVINO cmake script / infra label Mar 16, 2026

Fix remaining copyright headers

43c1bb3

svigh requested a review from a team as a code owner March 16, 2026 14:00

svigh enabled auto-merge March 16, 2026 14:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIT] EISW-204381: Add per output threshold#34352

[SIT] EISW-204381: Add per output threshold#34352
svigh wants to merge 13 commits intoopenvinotoolkit:masterfrom
svigh:EISW-204381_add_per_output_threshold

svigh commented Feb 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 26, 2026

Uh oh!

Copilot AI Feb 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		} catch (...) {
		// Not a single number, parse as key:value pairs

-    } catch (...) {
-        // Not a single number, parse as key:value pairs
+    } catch (const std::exception& e) {
+        // Not a single number. If there is no colon, we cannot interpret this as "layer:value".
+        if (str.find(':') == std::string::npos) {
+            std::cerr << "Warning: Failed to parse per-layer value '" << str
+                      << "' as a single number and no 'layer:value' pairs were found. "
+                      << "Using default value " << defaultValue << "." << std::endl;
+            return result;
+        }
+        // Otherwise, fall through and try to parse as key:value pairs.

Conversation

svigh commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

svigh commented Feb 26, 2026 •

edited

Loading