T5SpTokenizer class

Constructors

T5SpTokenizer.fromJson(Map<String, dynamic> raw)
factory
T5SpTokenizer.loadFile(String path)
Load a T5 tokenizer.json (HuggingFace tokenizers-lib format with model.type == "Unigram").
factory

Properties

eosId → int
no setter
hashCode → int
The hash code for this object.
no setterinherited
padId → int
no setter
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
unkId → int
no setter
vocabSize → int
no setter
vocabulary → List<String>
Convenience for tests: read-only view of the id -> piece table.
no setter

Methods

decode(List<int> ids, {bool skipSpecial = true}) → String
Decode a list of ids back into a string, dropping special tokens (<pad>, </s>, <unk>, <extra_id_*>) unless skipSpecial is false.
encode(String text, {bool addEos = true}) → List<int>
Encode text to a list of int piece ids.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited