BaseTokenizer class
BaseTokenizer class for tokenization and detokenization.
- Implementers
Constructors
Properties
- bosToken ↔ String
-
getter/setter pair
- bosTokenId ↔ int
-
getter/setter pair
- eosToken ↔ String
-
getter/setter pair
- eosTokenId ↔ int
-
getter/setter pair
- hashCode → int
-
The hash code for this object.
no setterinherited
-
i2s
↔ Map<
int, String> -
getter/setter pair
- length → int
-
no setter
- padToken ↔ String
-
getter/setter pair
- padTokenId ↔ int
-
getter/setter pair
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
-
s2i
↔ Map<
String, int> -
getter/setter pair
- unkToken ↔ String
-
getter/setter pair
- unkTokenId ↔ int
-
getter/setter pair
Methods
-
detokenize(
List< int> tokens) → String - Detokenizes the tokens.
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
textToTokens(
String text) → List< String> - Converts text to tokens.
-
tokenize(
String text, {bool bos = false, bool eos = false}) → List< int> - Tokenizes the text.
-
tokensToText(
List< String> tokens) → String - Converts tokens to text.
-
toString(
) → String -
A string representation of this object.
inherited
-
vocab(
) → List< String> - Returns the vocabulary tokens.
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited