MulticlassNms/MatrixNms: transformations and CPU implementation (openvinotoolkit#6653)

* init version, need revise: opset7

* add convert testcase

* multiclass_nms support spec

* init version

* matrixnms support spec

* init support for matrix_nms

* impl matirx_nms

* implemented multiclass_nms reference.

TODO: more test cases.

* support dynamic shape in test

* update to spec 0611

* update to spec 0611

* fixes.

* fix: now sort by class_id and score work.

* fix clang check error

* more test cases verified.

* fixes in ref impl.

* attribute nms_eta works

* test cross_batch and output_type i32.

* enable multiclass-nms cpu plugin fallback ngraph

* keep topk typo

* enable matrix-nms cpu plugin fallback ngraph

* support sort_result_across_batch

* Add matrix_nms unit test

* Add cross batch test cases

* fix typo

* move multiclass to opset8

* move matrixnms to opset8

* Reference implementations for MulticlassNms and MatrixNms ops

* fix name conflict

* remove unused var
sort_result_across_batch default set to false

* avoid float overflow

* fix clang check error

* info for mac fail

* change testcase due to unstable sort

* nms add 'normalized' attribute

* multiclass cpu test support 'normalized'

* nms add 'normalized' attribute

* fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch.

* fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch.

* fix sort order in matrix_nms

* fix review comments

* add matrix_nms MKLDNN extension layer

* parallel in matirx nms

* separate filtered_box

* separate class_nms result

* parallel in class

* parallel in batch

* partial new nms

* partial remove useless function

* debug & fix

* debug in indexing

* fix test cases

* remove logging

* fix code-style

* fix typo

* add matrix_nms extension

* nms python api

* remove unused testcases

* refactor transformation

* transform dynamic shape to static shape

* Update inference-engine/src/transformations/include/ngraph_ops/nms_static_shape_ie.hpp

Co-authored-by: Ilya Churaev <[email protected]>

* remove register_pass call

* [MKLDNN]migrate matrix_nms to MKLDNNNode

* bug fix in matrix_nms

* padding on matrix_nms

* remove logging

* test case refine

* merged transform_matrix_nms branch

* refine matrixnms testcase

* multiclass nms cpu plugin implement for static shape, rebased on Reference implementations PR

* rebase to new multi-classs transform provided by lc

* Name style algin with matrix-nms

* static shape padding style to batch inside,new unit test method, real classnum shape

* fix format

* fix ci error

* multi-class NMS modification based on PR reviewer opinion: code format, copyright, delete unused include and funciton way

* explicit template instantiation due to mac ci fail

* Yi3/fix review (#16)

* fix coding style

* use parallel_for2d

* fix ci fail

* unify 'copyright 2021'

* mkldnn_multiclass_nms node update based on PR review (#17)

* [MKLDNN] apply suggestion for matrix_nms (#18)

* fix bug

* apply review comments

* apply review comments

* apply review comments

* apply review comments

* skip only Nms test, not MatrixNms MulticlassNms test

Co-authored-by: Zhang Yi3 <[email protected]>
Co-authored-by: jialipen <[email protected]>
Co-authored-by: mangguo <[email protected]>
Co-authored-by: Ilya Churaev <[email protected]>
Co-authored-by: liubo-intel <[email protected]>

Author: 6 people

inference-engine/src/inference_engine/cnn_network_ngraph_impl.cpp

@@ -35,6 +35,9 @@
 
 #include <transformations/low_precision/disable_convert_constant_folding_on_const_path.hpp>
 
+#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
+#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
+
 #include "ie_ngraph_utils.hpp"
 #include "exec_graph_info.hpp"
 #include "ie_itt.hpp"
@@ -389,6 +392,8 @@ CNNNetworkNGraphImpl::reshape(const std::map<std::string, ngraph::PartialShape>&
                 ::ngraph::pass::Manager manager;
                 // resolves dynamism by replacing dynamic operation with static version
                 manager.register_pass<::ngraph::pass::ConvertNMS5ToLegacyMatcher>(false);
+                manager.register_pass<::ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
+                manager.register_pass<::ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
                 manager.register_pass<::ngraph::pass::DisableConvertConstantFoldingOnConstPath>();
                 manager.register_pass<::ngraph::pass::ConstantFolding>();
                 // OneHotToLegacy changes output precision

inference-engine/src/mkldnn_plugin/cpu_types.h

@@ -86,7 +86,9 @@ enum Type {
     ExperimentalDetectronPriorGridGenerator,
     ExperimentalDetectronGenerateProposalsSingleImage,
     ExtractImagePatches,
-    NonMaxSuppression
+    NonMaxSuppression,
+    MatrixNms,
+    MulticlassNms
 };
 
 enum Algorithm {

inference-engine/src/mkldnn_plugin/mkldnn_node.cpp

@@ -225,7 +225,9 @@ static const InferenceEngine::details::caseless_unordered_map<std::string, Type>
         { "ExperimentalDetectronPriorGridGenerator", ExperimentalDetectronPriorGridGenerator},
         { "ExperimentalDetectronGenerateProposalsSingleImage", ExperimentalDetectronGenerateProposalsSingleImage},
         { "ExtractImagePatches", ExtractImagePatches},
-        { "NonMaxSuppressionIEInternal", NonMaxSuppression}
+        { "NonMaxSuppressionIEInternal", NonMaxSuppression},
+        { "MatrixNms", MatrixNms},
+        { "MulticlassNms", MulticlassNms}
 };
 
 Type TypeFromName(const std::string type) {

inference-engine/src/mkldnn_plugin/mkldnn_node.h

@@ -194,6 +194,10 @@ static std::string NameFromType(Type type) {
             return "ExtractImagePatches";
         case NonMaxSuppression:
             return "NonMaxSuppression";
+        case MatrixNms:
+            return "MatrixNms";
+        case MulticlassNms:
+            return "MulticlassNms";
         default:
             return "Unknown";
     }

inference-engine/src/mkldnn_plugin/mkldnn_plugin.cpp

@@ -57,6 +57,8 @@
 #include <transformations/op_conversions/simplify_ctc_greedy_decoder_seq_len.hpp>
 #include <transformations/op_conversions/convert_previous_nms_to_nms_5.hpp>
 #include <transformations/op_conversions/convert_nms_to_nms_ie_internal.hpp>
+#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
+#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
 #include <transformations/op_conversions/convert_deformable_conv_v8_to_v1.hpp>
 #include <transformations/smart_reshape/matmul_sr.hpp>
 #include <transformations/convert_precision.hpp>
@@ -168,6 +170,8 @@ static void Transformation(CNNNetwork& clonedNetwork, const Config& conf) {
     manager.register_pass<ngraph::pass::ConvertNMS3ToNMS5>();
     manager.register_pass<ngraph::pass::ConvertNMS4ToNMS5>();
     manager.register_pass<ngraph::pass::ConvertNMSToNMSIEInternal>();
+    manager.register_pass<ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
+    manager.register_pass<ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
     manager.register_pass<ngraph::pass::TransposeMatMul>();
     manager.register_pass<ngraph::pass::ConstantFolding>();