EmbeddingTokenizerFactory typedef

EmbeddingTokenizerFactory = Future<EmbeddingTokenizer> Function(String tokenizerPath)

Factory that builds an EmbeddingTokenizer for a given on-disk tokenizer path.

MUST be a top-level or static function — a genuine tear-off — and never a closure or bound instance method, for exactly the reason documented on EmbeddingForwardPassFactory in forward_pass.dart: a ForwardPassDescriptor carrying this factory is sent through a SendPort into a freshly Isolate.spawn'd isolate (see test/forward_pass_isolate_test.dart for the boundary proof), and only a top-level/static tear-off survives that trip intact — a closure over local/instance state is either rejected outright or silently loses the captured object identity, and a runtime registry populated via some register() call is empty in the freshly spawned isolate no matter what the main isolate registered into it before spawning.

Implementation

typedef EmbeddingTokenizerFactory =
    Future<EmbeddingTokenizer> Function(String tokenizerPath);