core/embedding/tokenizer_adapter library

Classes

EmbeddingTokenizer
Turns raw text (with a TaskType prefix already applied) into a TokenizedInput. One instance per loaded tokenizer file; built once by the worker at startup and reused for every encode call.
TokenizedInput
One text's tokenized form, ready for EmbeddingForwardPass.run.

Typedefs

EmbeddingTokenizerFactory = Future<EmbeddingTokenizer> Function(String tokenizerPath)
Factory that builds an EmbeddingTokenizer for a given on-disk tokenizer path.