core/data/char_tokenizer library
Character-level tokenizer.
Builds its vocabulary from the sorted set of unique characters in
the training corpus (mirrors the reference dart_cuda tokenizer
used by the tiny-Shakespeare examples). Every char maps to a
dense integer id in [0, vocabSize).
Unknown characters at encode time map to id 0. decode treats
out-of-range ids as the empty string so it stays safe on
hallucinated model output.