StringTokenizer class

A tokenizer that breaks strings into lexical tokens for parsing.

StringTokenizer implements a lexer that scans source text and identifies tokens such as numbers, identifiers, operators, and punctuation. It maintains a current position and provides methods for consuming characters and patterns.

This is used by Parser to convert string expressions into parseable tokens before building unit objects.

Constructors

StringTokenizer(String source)
Creates a tokenizer for the entire source string.
StringTokenizer.fromRange(String source, int index, int endIndex)
Creates a tokenizer for a range within the source string.

Properties

endIndex → int
The end index (exclusive) for tokenization.
final
hashCode → int
The hash code for this object.
no setterinherited
index ↔ int
The current position in the source string.
getter/setter pair
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
source → String
The source string being tokenized.
final

Methods

eat(bool test(String char, int index), int? length) → ({int endIndex, int startIndex})?
Consumes characters that match a test function and returns their range.
eatBoolean() → bool?
Attempts to parse and consume a boolean value ('true' or 'false').
eatCharacter(String char) → bool
Attempts to consume a single char from the current position.
eatNumber() → double?
Attempts to parse and consume a number (integer or decimal) from the current position.
eatString(String string) → bool
Attempts to consume the exact string from the current position.
eatWhitespace() → bool
Consumes any whitespace characters from the current position.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
tokenize() → List<Token>
Tokenizes the source string into a list of tokens.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited