TokenizedInput class

One text's tokenized form, ready for EmbeddingForwardPass.run.

attentionMask / tokenTypeIds are null for tokenizers that don't produce them (Gemma SentencePiece); WordPiece tokenizers always populate them (all-ones mask, all-zeros type-ids for single-segment input).

Constructors

TokenizedInput({required List<int> ids, List<int>? attentionMask, List<int>? tokenTypeIds})
const

Properties

attentionMask → List<int>?
Attention mask aligned 1:1 with ids (1 = real token, 0 = padding). null when the tokenizer's convention has no notion of padding at this stage (e.g. Gemma SentencePiece — the forward pass pads/truncates internally and has no mask to report back either).
final
hashCode → int
The hash code for this object.
no setterinherited
ids → List<int>
Token ids, including any special tokens (BOS/EOS or CLS/SEP) the tokenizer's convention adds. Not yet padded/truncated to a fixed sequence length — that is the forward pass's job (see forward_pass.dart's module doc).
final
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
tokenTypeIds → List<int>?
Token-type (segment) ids aligned 1:1 with ids. null for tokenizers that don't use them (Gemma SentencePiece); all-zeros for single-segment WordPiece input.
final

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited