flutter_gemma_speech #

On-device speech for flutter_gemma — STT, TTS, and a VoiceSession voice loop — via the LiteRT C API + dart:ffi. Opt-in package: add it only if your app needs speech-to-text, text-to-speech, or a push-to-talk voice loop.

This package depends on flutter_gemma_litertlm, which owns the shared libLiteRtLm native bundle and exposes the LiteRt interpreter FFI (LiteRtBindings) used here.

Status #

• STT works end-to-end for moonshine-tiny (raw-PCM seq2seq), Whisper and Parakeet (log-mel) via LiteRtSttBackend.
• TTS works end-to-end for Matcha (litert-community/Matcha-TTS, 22050 Hz — a 3-graph LiteRT pipeline: encoder → CFM decoder → HiFi-GAN vocoder, producing 16-bit PCM), Qwen3-TTS and Inflect-Nano-v2 via LiteRtTtsBackend; kokoro/supertonic are follow-ons.

Both backends are pure factories (canHandle always true) — the model is selected per-install via SttModelType / TtsModelType, not the backend.

Usage #

import 'package:flutter_gemma/flutter_gemma.dart';
import 'package:flutter_gemma_speech/flutter_gemma_speech.dart';

await FlutterGemma.initialize(
  sttBackends: [LiteRtSttBackend()],
  ttsBackends: [LiteRtTtsBackend()],
);

// Text-to-speech (Matcha): install the bundle once, then synthesize.
await FlutterGemma.installTts()
    .fromNetwork('https://huggingface.co/litert-community/Matcha-TTS/resolve/main/')
    .ofType(TtsModelType.matcha)
    .install();

final synth = await FlutterGemma.getActiveTts();
final pcm = await synth.synthesize('Hello world.'); // Uint8List, 16-bit PCM
print(synth.sampleRate); // 22050
await synth.close();

Speech-to-text output language (Whisper) #

Whisper's shipped checkpoints are multilingual. The output language is one token in the decoder's seed prompt, rebuilt per transcription — so it is a per-call knob, and switching it never reloads the model:

final recognizer = await FlutterGemma.getActiveStt(language: 'de');
final german = await recognizer.transcribe(germanPcm);

// Same recognizer, one call in French.
final french = await recognizer.transcribe(frenchPcm, language: 'fr');

Codes are Whisper's own, without the delimiters ('en', 'de', 'uk'); default 'en'. The setting changes what the model WRITES, not what it hears — 'en' on German audio returns an English translation, not an error. moonshine and Parakeet have no language token and throw ArgumentError rather than ignoring the value.

Voice loop #

VoiceSession chains STT → LLM → TTS into one push-to-talk turn with barge-in. VoiceSession.fromChat wraps an InferenceChat. A chat with tools is supported — pass onToolCall and the turn runs through the tool loop; it throws only if a tools-enabled chat arrives without a handler. Use VoiceSession.custom for an AgentSession/MCP responder. runTurn takes recorded PCM and streams back VoiceEvents.

final recognizer = await FlutterGemma.getActiveStt();
final synthesizer = await FlutterGemma.getActiveTts();
final chat = await (await FlutterGemma.getActiveModel(maxTokens: 1024))
    .createChat(tokenBuffer: 256, maxOutputTokens: 128); // no tools, short replies

final session = VoiceSession.fromChat(
  recognizer: recognizer, chat: chat, synthesizer: synthesizer);

await for (final event in session.runTurn(pcm16kMono)) {
  switch (event) {
    case VoiceTranscriptEvent(:final text): /* show */
    case VoiceReplyTextEvent(:final chunk): /* stream */
    case VoiceReplyAudioEvent(:final pcm, :final sampleRate): /* play */
    case VoiceTurnInterruptedEvent(): /* stop player */
    case VoiceTurnCompleteEvent(): case VoiceErrorEvent(): break;
  }
}
// Barge-in: await session.interrupt();

VoiceSession owns no microphone or player — the app captures PCM (package:record) and plays the reply (pcmToWav + package:just_audio), exactly as the example stt_screen/tts_screen do.

Platforms #

No hook/build.dart of its own — the native library is bundled by flutter_gemma_litertlm's Native Assets hook and shared transitively.