Add missing HF functionalities (#170) (#17569)

Summary:
Pull Request resolved: #17569

* [FEAT] Add prepend normalizer

This commit introduces prepend normalizer, similiar to hugging faces's
rust implementation.

* [FEAT] Add "skip_special_tokens" parameter to the decode function

Added decode function parameter, to optionaly skip decoding special
tokens. Similiary to the HF Rust implementaiton. This change should be
agnostic unless set to true.

* [FEAT] Add funciton "piece_to_id"

This commit introduces public member function that converts string to
token id. This function is reverse of already existing 'id_to_piece'

* [FEAT] Add handling of null pretokenizer and bytefallback json fields

Added:
  - Handling of pretokenizer field explicitly set to null
  - Handling of bytefallback field along with encode logic

* [FIX] Changed decode API to work on vectors instead of singular tokens

* [REFACTOR] Changed tests to reflect new decode API

* [FIX] Change decoders to work on vectors

* [FEAT] Make postprocessing a separte step

Postprocessing is now separate, configurable step similiar to normalization, pretokenization or decoding.

* Revert "[FIX] Changed decode API to work on vectors instead of singular tokens"

This reverts commit 08e1b399e4fafcecc78c1941b6331782f7d65469.

* [REFACTOR] Split loading function of HFTokenizer

* Revert tests as there's no longer vectorized decode api

* [FIX] Fix handling of unknown tokens in bpm

* [FIX] Added FuseDecoder implementation

* Fix python bindings

* [FIX] post_processor, remove silent fails

This commit, removes BertProcessor and RobertaProcessor skeleton
classes.

* chore: Add test cases

This commit adds test cases for:
- PieceToId logic
- skip_special_tokens logic
- PrependNormalizer

* chore: add python binding for batch decode

* chore: add post_processor to BUCK file

* chore: fix formatting in token_decoder.h, remove placeholder code in post_processor.h

* chore: change copyright handle to SWM

* feat: add tests requested in review

* chore: Unify logs in piece_to_id definitions

* chore: fix tests to ensure parity with rust implementaiton outputs

* chore: add python test for batch_decode

Differential Revision: D93019471

Author: larryliu0820

extension/llm/runner/test/test_text_llm_runner.cpp

@@ -31,26 +31,28 @@ namespace {
 // Mock classes for dependencies
 class MockTokenizer : public ::tokenizers::Tokenizer {
  public:
-  MOCK_METHOD(::tokenizers::Error, load, (const std::string&), ());
-  MOCK_METHOD(bool, is_loaded, (), (const));
+  MOCK_METHOD(::tokenizers::Error, load, (const std::string&), (override));
+  MOCK_METHOD(bool, is_loaded, (), (const, override));
   MOCK_METHOD(
       ::tokenizers::Result<std::vector<uint64_t>>,
       encode,
       (const std::string&, int8_t, int8_t),
-      (const));
+      (const, override));
   MOCK_METHOD(
       ::tokenizers::Result<std::string>,
       decode,
-      (uint64_t, uint64_t),
-      (const));
+      (uint64_t, uint64_t, bool),
+      (const, override));
   MOCK_METHOD(
       ::tokenizers::Result<std::string>,
       id_to_piece,
       (uint64_t),
-      (const));
-  MOCK_METHOD(uint64_t, bos_tok, (), (const));
-  MOCK_METHOD(uint64_t, eos_tok, (), (const));
-  MOCK_METHOD(uint64_t, vocab_size, (), (const));
+      (const, override));
+  MOCK_METHOD(
+      ::tokenizers::Result<uint64_t>,
+      piece_to_id,
+      (const std::string&),
+      (const, override));
 };
 
 class MockModule : public ::executorch::extension::Module {
@@ -128,17 +130,17 @@ class RunnerTest : public Test {
               std::vector<uint64_t>{1, 2, 3});
         });
 
-    ON_CALL(*tokenizer, decode).WillByDefault([](uint64_t, uint64_t) {
+    ON_CALL(*tokenizer, decode).WillByDefault([](uint64_t, uint64_t, bool) {
       return ::tokenizers::Result<std::string>("token");
     });
 
     ON_CALL(*tokenizer, id_to_piece).WillByDefault([](uint64_t) {
       return ::tokenizers::Result<std::string>("piece");
     });
 
-    ON_CALL(*tokenizer, bos_tok()).WillByDefault(Return(1));
-    ON_CALL(*tokenizer, eos_tok()).WillByDefault(Return(2));
-    ON_CALL(*tokenizer, vocab_size()).WillByDefault(Return(100));
+    ON_CALL(*tokenizer, piece_to_id).WillByDefault([](const std::string&) {
+      return ::tokenizers::Result<uint64_t>(0);
+    });
 
     return tokenizer;
   }