Jinja Minimal #

A minimalistic pure Dart implementation of the Jinja templating engine, specifically designed for parsing and rendering ML chat templates.

Based off the fine work done by huggingface.js/jinja.

This uses no reflection (works in flutter!) and can even handle functions and Objects.

🗒️ Note:
Objects should implement a toJson method or be converted before calling render. Example method signature:

Map<String, dynamic> toJson() => throw UnimplementedError('TODO');

Which means this works great with the json_serializable package!

Usage #

Load template from a model on the Hugging Face Hub #

First, install the jinja_minimal and huggingface_hub packages:

flutter pub add jinja_minimal huggingface_hub

🗒️ Note:
While jinja_minimal is pure dart, huggingface_hub is not (at the time of writing) which is why flutter is used to install the packages.

You can then load a tokenizer from the Hugging Face Hub and render a list of chat messages, as follows:

import 'dart:convert';
import 'dart:io';

import 'package:huggingface_hub/huggingface_hub.dart';
import 'package:jinja_minimal/jinja_minimal.dart';

void main() async {
  final String configPath = await hfHubDownload(
    repoId: 'mistralai/Mistral-7B-Instruct-v0.1',
    filename: 'tokenizer_config.json',
  );

  final Map<String, dynamic> config = jsonDecode(
    await File(configPath).readAsString(),
  );

  const List<Map<String, dynamic>> chat = [
    { 'role': 'user', 'content': 'Hello, how are you?' },
    { 'role': 'assistant', 'content': "I'm doing great. How can I help you today?" },
    { 'role': 'user', 'content': "I'd like to show off how chat templating works!" },
  ];

  final template = Template(config['chat_template']);
  final result = template.render({
    'messages': chat,
    'bos_token': config['bos_token'],
    'eos_token': config['eos_token'],
  });

  // <s> [INST] Hello, how are you? [/INST] I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]
}

transformers_dart #

First, install transformers:

flutter pub add transformers

You can then render a list of chat messages using a tokenizer's apply_chat_template method.

import 'package:transformers/transformers.dart';

void main() async {
  // Load tokenizer from the Hugging Face Hub
  final tokenizer = await AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1");

  // Define chat messages
  const List<Message> chat = [
    Message(role: 'user', content: 'Hello, how are you?'),
    Message(role: 'assistant', content: "I'm doing great. How can I help you today?"),
    Message(role: 'user', content: "I'd like to show off how chat templating works!"),
  ];

  final String text = tokenizer.apply_chat_template(chat, ApplyChatTemplateOptions(
    tokenize: false,
  ));

  // <s> [INST] Hello, how are you? [/INST] I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]
}

Notice how the entire chat is condensed into a single string. If you would instead like to return the tokenized version (i.e., a list of token IDs), you can use the following:

final Tensor input_ids = tokenizer.apply_chat_template(chat, ApplyChatTemplateOptions(
  tokenize: true,
  return_tensor: false,
));
// [1, 733, 16289, 28793, 22557, 28725, 910, 460, 368, 28804, 733, 28748, 16289, 28793, 28737, 28742, 28719, 2548, 1598, 28723, 1602, 541, 315, 1316, 368, 3154, 28804, 2, 733, 16289, 28793, 315, 28742, 28715, 737, 298, 1347, 805, 910, 10706, 5752, 1077, 3791, 28808, 733, 28748, 16289, 28793]

For more information about chat templates, check out the transformers documentation.