token_text_controller #

A token-aware TextEditingController for Flutter. It detects and styles semantic text while preserving a normal editable string, cursor, selection, clipboard behavior, and IME composition.

Use it with any TextField, TextFormField, or EditableText. No custom input widget is required.

Features #

• Every built-in detector enabled by default, with fine-grained activation.
• Reusable token groups for messaging, social, contact, commerce, structured data, suggestions, links, and Markdown.
• Stable light/dark token colors independent of theme brand colors.
• A dedicated optional TextStyle for every built-in token kind.
• Regex, function, trigger, and fully custom token detectors.
• Priority-based overlap resolution.
• Unicode grapheme handling for joined, modified, keycap, and flag emoji.
• Current-token queries for mention, hashtag, command, and cashtag suggestions.
• Token list, newly detected token, and active query callbacks.
• Semantic token insertion with arbitrary custom metadata.
• A reusable TokenText renderer for non-editable content.
• Correct composing-range rendering for international keyboards and IMEs.

Installation #

Add the published package to pubspec.yaml:

dependencies:
  token_text_controller: ^0.3.1

Minimal usage #

final controller = TokenTextController();

TextField(
  controller: controller,
  decoration: const InputDecoration(hintText: 'Message'),
)

By default, every built-in token kind is active, including links, contact values, social tokens, structured values, lightweight Markdown, commands, and emoji. controller.text remains an ordinary string.

Markdown rendering #

Recognized inline formatting renders without its delimiters by default:

final controller = TokenTextController(
  text: '**Important** _note_ ~~old~~ ||spoiler|| `code` [Flutter](https://flutter.cn)',
);

Bold supports **text** and __text__; italic supports _text_ and *text*. Markdown links show only their label. Their destination is available in token.data; token.displayText and token.contentRange expose the visible portion. token.text, token.range, controller.text, and serialized documents always retain the original source, including syntax.

In an editable field, hidden syntax has zero visual width but keeps its UTF-16 positions. It is not deleted or replaced by widget placeholders, so selections and the IME keep working with the original value. Composing ranges underline the visible content without exposing the hidden syntax. Copying from the editable field keeps the source; read-only/selectable TokenText copies rendered text.

Use showMarkdownSyntax: true in either constructor to show delimiters. The controller property can also change while editing:

controller.showMarkdownSyntax = true;

Disabling Markdown token kinds leaves their syntax literal. Inline code does not interpret nested links or formatting; escaped and incomplete markers are not treated as formatting. Prefixes such as @, #, $, and template braces remain visible. These are lightweight, non-overlapping inline detectors, not a full CommonMark/block editor. Formatting-like punctuation inside detected URLs, email addresses, handles, hashtags, and variables stays literal. TokenTextSpanBuilder preserves source length by default; only read-only consumers should set preserveSourceLength: false.

Styling #

Every built-in token accepts a complete Flutter TextStyle. Unspecified styles fall back to theme-aware defaults.

The default palette adapts to light/dark brightness without using the primary, secondary, or tertiary brand colors:

Blue follows the familiar link convention. The other colors are package defaults, not universal token standards. Links and contact details are underlined; mentions and hashtags have extra weight. Surfaces, ordinary text, and code text still follow ColorScheme. Check contrast when using custom backgrounds.

Emoji use a stable 20 font size by default in both mixed text and emoji-only values. Set emoji or emojiOnly to customize the size.

final controller = TokenTextController(
  styles: TokenTextStyles(
    base: TextStyle(fontSize: 16),
    url: TextStyle(decorationThickness: 2),
    email: TextStyle(fontWeight: FontWeight.w500),
    mention: TextStyle(
      color: Colors.blue,
      fontWeight: FontWeight.w700,
    ),
    hashtag: TextStyle(color: Colors.green),
    command: TextStyle(color: Colors.deepPurple),
    emoji: TextStyle(fontSize: 20),
    emojiOnly: TextStyle(fontSize: 30),
  ),
);

TokenTextStyles also provides styles for phone numbers, references, dates, times, currencies, percentages, UUIDs, IP addresses, variables, Markdown links, inline code, bold, italic, strikethrough, spoilers, and custom kinds.

Use TokenTextStyles.unstyled() when no built-in visual treatment is desired. For per-value or metadata-driven styling, provide a resolver:

TokenTextController(
  styleResolver: (details) {
    if (details.token.data == 'verified') {
      return details.suggestedStyle.copyWith(
        decoration: TextDecoration.underline,
      );
    }
    return details.suggestedStyle;
  },
)

Suggestions and active queries #

onActiveTokenChanged and activeQueryListenable report the trigger token at the caret, including incomplete values such as @lu, #flu, or /he.

final controller = TokenTextController(
  onActiveTokenChanged: (query) {
    if (query?.kind == TokenKind.mention) {
      loadProfiles(query!.query);
    }
  },
);

A suggestion can replace the complete active range and retain domain data:

controller.replaceActiveQuery(
  '@Luis',
  kind: TokenKind.mention,
  data: {'profileId': 'profile-123'},
);

The inserted token appears in controller.semanticTokens and remains linked to its metadata until an edit intersects that token.

Serialization #

Persist plain text and resolved semantic metadata together:

final json = controller.document.toJson();
final document = TokenTextDocument.fromJson(json);
final restoredController = TokenTextController.fromDocument(document);

Use encodeData and decodeData when token metadata needs conversion to a JSON-compatible representation.

Token events #

TokenTextController(
  onTokensChanged: (tokens) {
    // Current non-overlapping token snapshot.
  },
  onTokenDetected: (token) {
    // A token has newly appeared in the editable value.
  },
  onActiveTokenChanged: (query) {
    // The token query at the current collapsed selection changed.
  },
)

The same state is available through tokensListenable and activeQueryListenable for widget composition without callbacks.

Active tokens and groups #

Use activeTokens to enable only the token kinds needed by a field. The default is TokenGroups.all.

final controller = TokenTextController(
  activeTokens: TokenGroups.chat,
);

Groups are intentionally scoped to a concrete kind of field:

Groups are regular constant lists, so they can be used directly or combined:

final controller = TokenTextController(
  activeTokens: TokenGroups.combine(
    [TokenGroups.social, TokenGroups.markdown],
    include: const [TokenKind.reference],
    exclude: const [TokenKind.cashtag],
  ),
);

Activation can change while the field is alive. Existing text, styles, token events, and the active suggestion query are recalculated immediately.

controller.activeTokens = TokenGroups.suggestions;
controller.activateTokens(const [TokenKind.emoji]);
controller.deactivateTokens(const [TokenKind.cashtag]);
controller.activateAllTokens();

availableTokens reports the kinds backed by the controller's detector catalog, while isTokenActive(kind) checks the current selection.

Detector presets #

TokenTextController(detectors: TokenDetectors.social());
TokenTextController(detectors: TokenDetectors.chat());
TokenTextController(detectors: TokenDetectors.messaging());
TokenTextController(detectors: TokenDetectors.assistantChat());
TokenTextController(detectors: TokenDetectors.commerce());
TokenTextController(detectors: TokenDetectors.structured());
TokenTextController(detectors: TokenDetectors.markdown());
TokenTextController(detectors: TokenDetectors.all());

Use detectors to replace or customize the detector catalog. All supplied detectors are active unless activeTokens is also provided. TokenDetectors also exposes forKind and forKinds for building a catalog from token kinds.

Custom tokens #

Token kinds are value objects rather than a closed enum:

const orderKind = TokenKind('order');

final controller = TokenTextController(
  detectors: [
    ...TokenDetectors.defaults(),
    RegexTokenDetector(
      kind: orderKind,
      id: 'order',
      priority: 150,
      pattern: RegExp(r'ORD-\d+'),
    ),
  ],
  styles: TokenTextStyles(
    custom: {
      orderKind: TextStyle(
        color: Colors.orange,
        fontWeight: FontWeight.bold,
      ),
    },
  ),
);

Implement TokenDetector directly or use FunctionTokenDetector when a regular expression is not sufficient. Higher priorities win when ranges overlap.

Rendering outside a field #

TokenText(
  message,
  activeTokens: TokenGroups.chat,
  styles: controller.styles,
  semanticTokens: controller.semanticTokens,
)

This package performs token-aware styling and semantic range management. It is not a document editor and does not alter the source string with hidden markup.