Platformsfold toggle (up/down arrow)

SDKsfold toggle (up/down arrow)

Licensefold toggle (up/down arrow)

Advancedfold toggle (up/down arrow)

Results 11 packages
Sort by

dart_sentencepiece_tokenizercopy "dart_sentencepiece_tokenizer: ^1.4.1" to clipboard
dart_sentencepiece_tokenizer: ^1.4.1 copied to clipboard

6
likes
160
points
24.4k
downloads
A lightweight pure Dart SentencePiece tokenizer supporting BPE (Gemma), Unigram (Llama), and Hugging Face tokenizer.json pipelines.#nlp#sentencepiece#tokenizer#machine-learning#llm

v 1.4.1 (7 days ago)verified publisherbrodykim.workMIT

tiktoken_tokenizer_gpt4o_o1copy "tiktoken_tokenizer_gpt4o_o1: ^1.2.1" to clipboard
tiktoken_tokenizer_gpt4o_o1: ^1.2.1 copied to clipboard

3
likes
140
points
3.1k
downloads
OpenAI's Tiktoken tokenizer for models: GPT-4, GPT-4o, GPT-4o-mini, o1, o1-mini, and o1-preview.#tokenizer#openai#gpt#gpt-4o#tiktoken

v 1.2.1 (17 months ago)verified publisherglasberg.devBSD-2-Clause

tiny_segmenter_dartcopy "tiny_segmenter_dart: ^1.0.1" to clipboard
tiny_segmenter_dart: ^1.0.1 copied to clipboard

4
likes
135
points
153
downloads
A compact Japanese text tokenizer for Dart. TinySegmenter is a Japanese word segmentation library based on the original JavaScript implementation by Taku Kudo.#japanese#text-processing#tokenizer#nlp#segmentation

v 1.0.1 (15 months ago)verified publisheriori.devunknown

model2veccopy "model2vec: ^2.0.3" to clipboard
model2vec: ^2.0.3 copied to clipboard

2
likes
160
points
189
downloads
On-device Model2Vec text embeddings for Dart & Flutter — a self-contained Rust core via FFI and Native Assets. Fast, local, static, minimal memory.#rag#nlp#embeddings#tokenizer#model2vec

v 2.0.3 (30 days ago)verified publisherpro100.devMIT

hf_tokenizerscopy "hf_tokenizers: ^1.2.1" to clipboard
hf_tokenizers: ^1.2.1 copied to clipboard

1
likes
160
points
231
downloads
HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.#llm#ai#tokenizer#ffi#nlp

v 1.2.1 (15 days ago)verified publisherdeveloperyusuf.comMIT

screenshot

dart_bert_tokenizercopy "dart_bert_tokenizer: ^1.0.2" to clipboard
dart_bert_tokenizer: ^1.0.2 copied to clipboard

1
likes
150
points
348
downloads
A lightweight, pure Dart implementation of BERT WordPiece tokenizer. 100% compatible with HuggingFace tokenizers.#nlp#bert#tokenizer#machine-learning#wordpiece

v 1.0.2 (8 months ago)verified publisherbrodykim.workMIT

betto_icucopy "betto_icu: ^0.1.0" to clipboard
betto_icu: ^0.1.0 copied to clipboard

0
likes
160
points
409
downloads
Unicode text tokenization for Dart — Tokenizer interface, IcuTokenizer (system ICU FFI, UAX #29), and RegExpTokenizer (pure Dart, Latin fallback). #text#unicode#nlp#tokenizer#icu

v 0.1.0 (20 days ago)verified publisherbettongia.comApache-2.0

rumil_tokenscopy "rumil_tokens: ^0.11.0" to clipboard
rumil_tokens: ^0.11.0 copied to clipboard

0
likes
160
points
228
downloads
Lossless source code tokenizer built on Rumil parser combinators. Classified token spans for syntax highlighting. Built-in grammars for Dart, Scala, YAML, JSON, and shell.#parser#tokenizer#syntax-highlighting#lexer

v 0.11.0 (7 days ago)verified publisherardaproject.orgMIT

flutter_syntax_highlightcopy "flutter_syntax_highlight: ^0.2.0" to clipboard
flutter_syntax_highlight: ^0.2.0 copied to clipboard

recently created package Added 7 days ago
0
likes
160
points
162
downloads
Dart-only syntax highlighting in two layers: a pure-Dart tokenizer underneath and a thin widget on top. The tokens rejoin into the exact input, byte for byte.#syntax-highlighting#tokenizer#code#text#widget

v 0.2.0 (6 days ago)verified publisherkihyun1998.comMIT

screenshot

token_countercopy "token_counter: ^1.0.0" to clipboard
token_counter: ^1.0.0 copied to clipboard

0
likes
160
points
39
downloads
Token counter for popular LLMs (OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama) with multi-language support. Pure Dart, works on all Flutter platforms and Dart VM.#llm#tokenizer#openai#anthropic#gemini

v 1.0.0 (4 months ago)verified publisherwarashibetech.comMIT

Check our help page for details on search expressions and result ranking.