CoreBPE class
BPE tokeniser
Read more on Wikipedia
Constructors
Properties
-
decoder
→ HashMap<
int, ByteArray> -
final
-
encoder
→ HashMap<
ByteArray, int> -
final
- hashCode → int
-
The hash code for this object.
no setterinherited
- regex → RegExp
-
final
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
-
sortedTokenBytes
→ List<
ByteArray> -
final
- specialRegex → RegExp
-
final
-
specialTokensDecoder
→ HashMap<
int, ByteArray> -
final
-
specialTokensEncoder
→ HashMap<
String, int> -
final
Methods
-
decodeNative(
List< int> tokens) → ByteArray -
decodeSingleTokenBytes(
int token) → ByteArray -
encodeNative(
String text, Set< String> allowedSpecial) → _Tuple2<Uint32List, int> -
encodeOrdinaryNative(
String text) → Uint32List -
encodeSingleToken(
ByteArray bytes) → int -
encodeUnstableNative(
String text, Set< String> allowedSpecial) → _Tuple2<List< int> , Set<List< >int> > -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenByteValues(
) → List< Uint8List> -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited