Skip to content

Fixes in tokenize docs #3756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+8 −8
Open

Fixes in tokenize docs #3756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b1b2457
docs fix, remove unused variable
przepeck Oct 31, 2025
697f4e9
removed deleted parameter
przepeck Oct 31, 2025
d670168
Merge branch 'main' into przepeck/tokenize_docs_fix
przepeck Oct 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions demos/embeddings/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,7 +510,7 @@ mteb run -m thenlper/gte-small -t Banking77Classification --output_folder result

# Usage of tokenize endpoint (release 2025.4 or weekly)

The `tokenize` endpoint provides a simple API for tokenizing input text using the same tokenizer as the deployed embeddings model. This allows you to see how your text will be split into tokens before feature extraction or inference. The endpoint accepts a string or list of strings and returns the corresponding token IDs and tokenized text.
The `tokenize` endpoint provides a simple API for tokenizing input text using the same tokenizer as the deployed embeddings model. This allows you to see how your text will be split into tokens before feature extraction or inference. The endpoint accepts a string or list of strings and returns the corresponding token IDs.

Example usage:
```console
Expand All @@ -524,10 +524,10 @@ Response:
```

It's possible to use additional parameters:
- pad_to_max_length - whether to pad the sequence to the maximum length. Default is False.
- max_length - maximum length of the sequence. If None (default), the value will be taken from the IR (where default value from original HF/GGUF model is stored).
- padding_side - side to pad the sequence, can be ‘left’ or ‘right’. Default is None.
- add_special_tokens - whether to add special tokens like BOS, EOS, PAD. Default is True.
- `pad_to_max_length` - whether to pad the sequence to the maximum length. Default is False.
- `max_length` - maximum length of the sequence. If None (default), unlimited.
- `padding_side` - side to pad the sequence, can be ‘left’ or ‘right’. Default is None.
- `add_special_tokens` - whether to add special tokens like BOS, EOS, PAD. Default is True.

Example usage:
```console
Expand Down
6 changes: 3 additions & 3 deletions src/embeddings/embeddings_calculator_ov.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ class EmbeddingsCalculatorOV : public CalculatorBase {

mediapipe::Timestamp timestamp{0};

absl::Status tokenizeStrings(ov::genai::Tokenizer& tokenizer, const std::vector<std::string>& inputStrings, const ov::AnyMap& parameters, ov::genai::TokenizedInputs& tokens, const size_t& max_context_length) {
absl::Status tokenizeStrings(ov::genai::Tokenizer& tokenizer, const std::vector<std::string>& inputStrings, const ov::AnyMap& parameters, ov::genai::TokenizedInputs& tokens) {
tokens = tokenizer.encode(inputStrings, parameters);
RET_CHECK(tokens.input_ids.get_shape().size() == 2);

Expand Down Expand Up @@ -134,7 +134,7 @@ class EmbeddingsCalculatorOV : public CalculatorBase {
}
auto input = tokenizeRequest.input;
if (auto strings = std::get_if<std::vector<std::string>>(&input)) {
auto tokenizationStatus = this->tokenizeStrings(embeddings_session->getTokenizer(), *strings, tokenizeRequest.parameters, tokens, max_context_length);
auto tokenizationStatus = this->tokenizeStrings(embeddings_session->getTokenizer(), *strings, tokenizeRequest.parameters, tokens);
if (!tokenizationStatus.ok()) {
return tokenizationStatus;
}
Expand Down Expand Up @@ -172,7 +172,7 @@ class EmbeddingsCalculatorOV : public CalculatorBase {
params["max_length"] = max_context_length;
}

absl::Status tokenizationStatus = this->tokenizeStrings(embeddings_session->getTokenizer(), *strings, params, tokens, max_context_length);
absl::Status tokenizationStatus = this->tokenizeStrings(embeddings_session->getTokenizer(), *strings, params, tokens);
if (!tokenizationStatus.ok()) {
return tokenizationStatus;
}
Expand Down