EmbeddingTokenizerProvider class abstract

A pluggable source of EmbeddingTokenizerFactory — the "text -> token ids" half of an embedding.

Same probe-chain shape as EmbeddingBackendProvider, and it exists for the same reason: which tokenizer a model needs is a property of the MODEL, not of the engine that runs it. EmbeddingGemma needs SentencePiece whether its weights are executed by LiteRT or by ONNX Runtime; MiniLM needs WordPiece either way. An engine that named a tokenizer would be asserting something it cannot know, and — because the implementations pull dart_sentencepiece_tokenizer — would have to depend on the package that owns them, which is the sibling edge Packages -> core, never to each other forbids.

So the engine asks the registry instead, and the app supplies the implementation:

FlutterGemma.initialize(
  embeddingBackends: [LiteRtEmbeddingBackend()],
  embeddingTokenizers: [GemmaEmbeddingTokenizers()],
);

An app that never embeds anything passes neither and pays for neither.