EmbeddingTokenizer class abstract
Turns raw text (with a TaskType prefix already applied) into a
TokenizedInput. One instance per loaded tokenizer file; built once by
the worker at startup and reused for every encode call.
Constructors
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
encode(
String prefix, String text) → TokenizedInput -
Tokenizes (
prefix+text) per this tokenizer's convention (BOS/EOS wrap for SentencePiece,CLS/.../SEPfor WordPiece). -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited