feat: implement ngram tokenizer with token_chars and custom_token_chars

[SpadeA-Tang]

issue: #45039

[sre-ci-robot]

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

• Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
• Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
• For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

• /ci-rerun-code-check // for ci-v2/code-check
• /ci-rerun-build // for ci-v2/build
• /ci-rerun-ut-integration // for ci-v2/ut-integration
• /ci-rerun-ut-go // for ci-v2/ut-go
• /ci-rerun-ut-cpp // for ci-v2/ut-cpp
• /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp

If you have any questions or requests, please contact @zhikunyao.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.93%. Comparing base (314ee81) to head (7a61a89).
⚠️ Report is 54 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #45040      +/-   ##
==========================================
- Coverage   76.97%   76.93%   -0.04%     
==========================================
  Files        1838     1838              
  Lines      285526   285503      -23     
==========================================
- Hits       219775   219648     -127     
- Misses      58555    58647      +92     
- Partials     7196     7208      +12     

Components	Coverage Δ
Client	78.03% <ø> (ø)
Core	83.88% <ø> (-0.01%)	⬇️
Go	75.03% <ø> (-0.06%)	⬇️
see 23 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zhengbuqian]

please cp 2.6

[SpadeA-Tang]

/ci-rerun-ut-go

[SpadeA-Tang]

please cp 2.6

done #45046

[SpadeA-Tang]

/ci-rerun-ut-go

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SpadeA-Tang, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/core/OWNERS [zhengbuqian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zhagnlu]

/lgtm