dart_sentencepiece_tokenizer 1.1.0 copy "dart_sentencepiece_tokenizer: ^1.1.0" to clipboard
dart_sentencepiece_tokenizer: ^1.1.0 copied to clipboard

A lightweight, pure Dart implementation of SentencePiece tokenizer. Supports BPE (Gemma) and Unigram (Llama) algorithms.

160/ 160
pub points
92
downloads

We analyzed this package 7 hours ago, and awarded it 160 pub points (of a possible 160):

Passed report section
Follow Dart file conventions
30 / 30trigger folding of the section

Passed check 10/10 points: Provide a valid pubspec.yaml

Passed check 5/5 points: Provide a valid README.md

Passed check 5/5 points: Provide a valid CHANGELOG.md

Passed check 10/10 points: Use an OSI-approved license

Detected license: MIT.

Passed report section
Provide documentation
20 / 20trigger folding of the section

Passed check 10/10 points: 20% or more of the public API has dartdoc comments

35 out of 107 API elements (32.7 %) have documentation comments.

Some symbols that are missing documentation: dart_sentencepiece_tokenizer.Encoding, dart_sentencepiece_tokenizer.Encoding.Encoding.empty, dart_sentencepiece_tokenizer.Encoding.Encoding.new, dart_sentencepiece_tokenizer.Encoding.attentionMask, dart_sentencepiece_tokenizer.Encoding.charToToken.

Passed check 10/10 points: Package has an example

Passed report section
Platform support
20 / 20trigger folding of the section

Passed check 20/20 points: Supports 5 of 6 possible platforms (iOS, Android, Web, Windows, macOS, Linux)

  • ✓ Android

  • ✓ iOS

  • ✓ Windows

  • ✓ Linux

  • ✓ macOS

These platforms are not supported:

Package not compatible with platform Web

Because:

  • package:dart_sentencepiece_tokenizer/dart_sentencepiece_tokenizer.dart that imports:
  • package:dart_sentencepiece_tokenizer/src/sentencepiece/sentencepiece_tokenizer.dart that imports:
  • package:dart_sentencepiece_tokenizer/src/sentencepiece/model/sentencepiece_model.dart that imports:
  • dart:io
Passed report section
Pass static analysis
50 / 50trigger folding of the section

Passed check 50/50 points: code has no errors, warnings, lints, or formatting issues

Passed report section
Support up-to-date dependencies
40 / 40trigger folding of the section

Passed check 10/10 points: All of the package dependencies are supported in the latest version

No dependencies.

To reproduce run dart pub outdated --no-dev-dependencies --up-to-date --no-dependency-overrides.

Passed check 10/10 points: Package supports latest stable Dart and Flutter SDKs

Passed check 20/20 points: Compatible with dependency constraint lower bounds

pub downgrade does not expose any static analysis error.

Analyzed with Pana 0.23.5, Dart 3.11.0-200.1.beta.

Check the analysis log for details.

Weekly downloads

Display as:
By versions:
0
likes
160
points
92
downloads

Publisher

verified publisherbrodykim.work

Weekly Downloads

A lightweight, pure Dart implementation of SentencePiece tokenizer. Supports BPE (Gemma) and Unigram (Llama) algorithms.

Repository (GitHub)
View/report issues

Topics

#nlp #sentencepiece #tokenizer #machine-learning #llm

Documentation

API reference

License

MIT (license)

More

Packages that depend on dart_sentencepiece_tokenizer