Bumping tokenizers pin to 1c432479f6b29ce8defc5d2e83375be9238fe1bf

[larryliu0820]

So that we can fix a test failure

Summary:

• [FEAT] Add prepend normalizer

This commit introduces prepend normalizer, similiar to hugging faces's
rust implementation.

• [FEAT] Add "skip_special_tokens" parameter to the decode function

Added decode function parameter, to optionaly skip decoding special
tokens. Similiary to the HF Rust implementaiton. This change should be
agnostic unless set to true.

• [FEAT] Add funciton "piece_to_id"

This commit introduces public member function that converts string to
token id. This function is reverse of already existing 'id_to_piece'

• [FEAT] Add handling of null pretokenizer and bytefallback json fields

Added:

• Handling of pretokenizer field explicitly set to null
• Handling of bytefallback field along with encode logic

• [FIX] Changed decode API to work on vectors instead of singular tokens

• [REFACTOR] Changed tests to reflect new decode API

• [FIX] Change decoders to work on vectors

• [FEAT] Make postprocessing a separte step

Postprocessing is now separate, configurable step similiar to normalization, pretokenization or decoding.

• Revert "[FIX] Changed decode API to work on vectors instead of singular tokens"

This reverts commit 08e1b399e4fafcecc78c1941b6331782f7d65469.

• [REFACTOR] Split loading function of HFTokenizer

• Revert tests as there's no longer vectorized decode api

• [FIX] Fix handling of unknown tokens in bpm

• [FIX] Added FuseDecoder implementation

• Fix python bindings

• [FIX] post_processor, remove silent fails

This commit, removes BertProcessor and RobertaProcessor skeleton
classes.

• chore: Add test cases

This commit adds test cases for:

• PieceToId logic
• skip_special_tokens logic
• PrependNormalizer

• chore: add python binding for batch decode

• chore: add post_processor to BUCK file

• chore: fix formatting in token_decoder.h, remove placeholder code in post_processor.h

• chore: change copyright handle to SWM

• feat: add tests requested in review

• chore: Unify logs in piece_to_id definitions

• chore: fix tests to ensure parity with rust implementaiton outputs

• chore: add python test for batch_decode

Differential Revision: D93019471

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17569

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures

As of commit c54996a with merge base a398a96 ():

NEW FAILURES - The following jobs have failed:

• pull / test-mediatek-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 72c0eca1376ed93fb21fd17373083788c014d0ce94715dbc899af97ba4283247 /exec failed with exit code 2
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t da632bbb5c0fe0aec96429cea86fca91311e3f6d665819c82d02d23b7b3d3f15 /exec failed with exit code 1
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 60e7546af6271fa381fbc5b3a16ceaa14e7ea5b609e8645e374317d3e381b851 /exec failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 8b261674537e937e6053ae0960efb7f6153d4ea4be5249cc23535feaa4a36214 /exec failed with exit code 3
• pull / unittest-buck / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 3

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@larryliu0820 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D93019471.