llm_sdk #

The unified toolkit for talking to any LLM from a Dart / Flutter app. One interface, several interchangeable brains.

On the web (JS) there is a polished toolkit for talking to LLMs (think Vercel AI SDK). On the Dart / Flutter side, there was nothing equivalent — everyone re-plumbs their own HTTP calls. llm_sdk is that clean bridge: multi-provider, streaming, tool calling and structured outputs behind one API.

Switching AI = changing one line #

Streaming, word by word #

Table of contents #

• Why llm_sdk
• Install
• Quick start
• Providers & configuration
• Local models (Ollama, LM Studio, llama.cpp, vLLM)
• The 4 building blocks
  • Text generation
  • Streaming
  • Tool calling
  • Structured outputs
• Sampling options
• Escape hatches
• Retries & timeouts
• Error handling
• Resource cleanup
• API reference
• Architecture
• Status & limitations
• Testing
• License

Why llm_sdk #

The contract a provider must implement is just two methods (generate, generateStream). All the usage logic — the tool loop, streamText, generateObject — is built once in LlmClient, on top of that contract. Switching from Claude to OpenAI to Gemini changes a single line; nothing else in your code moves.

Install #

Add the dependency:

dart pub add llm_sdk

or, in a Flutter project:

flutter pub add llm_sdk

or add it manually to your pubspec.yaml:

dependencies:
  llm_sdk: ^0.7.0

Then import it:

import 'package:llm_sdk/llm_sdk.dart';

The only runtime dependency is http, so the package works anywhere Dart runs (CLI, server, Flutter mobile/desktop/web). Requires Dart 3.4 or newer.

Quick start #

import 'dart:io';
import 'package:llm_sdk/llm_sdk.dart';

Future<void> main() async {
  // Pick the brain. Switching AI = changing this one line.
  final client = LlmClient(
    ClaudeProvider(apiKey: Platform.environment['ANTHROPIC_API_KEY']!),
  );

  final answer = await client.generateText([
    Message.system('You are a concise assistant.'),
    Message.user('Give me one productivity tip.'),
  ]);

  print(answer);
}

Tip: never hard-code API keys. Read them from environment variables (Platform.environment['...']) or your app's secret storage.

Providers & configuration #

Every provider implements the same LlmProvider contract, so they are fully interchangeable inside an LlmClient. They differ only in their constructor options and default model.

Claude (Anthropic) #

final provider = ClaudeProvider(
  apiKey: 'sk-ant-...',          // required
  model: 'claude-opus-5',        // default
  maxTokens: 1024,               // default — required by Anthropic
);

OpenAI (and OpenAI-compatible servers) #

final provider = OpenAIProvider(
  apiKey: 'sk-...',                       // optional — empty for local servers
  model: 'gpt-5.6-terra',                 // default
  maxTokens: null,                        // optional — null lets the model decide
  baseUrl: 'https://api.openai.com/v1',   // default — override for local models
);

Gemini (Google) #

final provider = GeminiProvider(
  apiKey: 'AIza...',                                            // required
  model: 'gemini-3.8-flash',                                    // default
  maxTokens: null,                                              // optional
  baseUrl: 'https://generativelanguage.googleapis.com/v1beta', // default
);

Common options #

Swapping providers #

Because the surface is identical, swapping is a one-line change:

final client = LlmClient(ClaudeProvider(apiKey: myKey));
// ... or, without touching any other line:
final client = LlmClient(OpenAIProvider(apiKey: myKey));
final client = LlmClient(GeminiProvider(apiKey: myKey));

You can also tune the tool loop bound:

final client = LlmClient(provider, maxSteps: 8); // default is 5

Local models (Ollama, LM Studio, llama.cpp, vLLM) #

Any server that exposes an OpenAI-compatible API works out of the box: just point baseUrl at your local endpoint. The API key is optional — a local server ignores it.

final client = LlmClient(OpenAIProvider(
  baseUrl: 'http://localhost:11434/v1', // Ollama
  model: 'llama3.2',
));

All 4 building blocks (text, streaming, tool calling, structured outputs) stay identical — only the baseUrl changes. No data leaves the machine.

Common endpoints:

The 4 building blocks #

1. Text generation #

generate returns a full LlmResponse (text, usage, finish reason, tool calls). generateText is the shortcut that returns the final string directly.

final response = await client.generate([
  Message.user('Summarize relativity in one sentence.'),
]);

print(response.text);                       // the answer
print(response.usage?.totalTokens);         // token count, if provided
print(response.finishReason);               // FinishReason.stop, length, ...

Build conversations by stacking messages:

final messages = [
  Message.system('You are a helpful translator.'),
  Message.user('Translate "good morning" to French.'),
  Message.assistant('Bonjour.'),
  Message.user('And to Spanish?'),
];
final reply = await client.generateText(messages);

2. Streaming #

Stream the answer word by word for a typewriter effect:

await for (final chunk in client.streamText([Message.user('Tell me a joke')])) {
  stdout.write(chunk); // typewriter effect
}

Need the raw event stream (text deltas, tool calls, and the assembled final response)? Use streamEvents:

await for (final event in client.streamEvents([Message.user('Hi')])) {
  switch (event) {
    case TextDelta(:final text):       stdout.write(text);
    case ToolCallDelta(:final call):   print('tool: ${call.name}');
    case StreamDone(:final response):  print('\nusage: ${response.usage}');
  }
}

Streaming with tools

Both streaming methods take tools and run the same automatic tool loop as generate. The model can narrate, ask for a tool, and keep streaming its final answer — all inside one stream:

await for (final chunk in client.streamText(
  [Message.user('Compare the weather in Douala and Yaoundé.')],
  tools: [weather], // ← runs automatically, mid-stream
)) {
  stdout.write(chunk);
}

TextDelta and ToolCallDelta are forwarded for every step, so a UI can show "running getWeather…" as it happens. StreamDone is emitted once, at the very end — never per step — and its usage is the sum across all steps, so a tool round-trip doesn't go uncounted (generate sums the same way).

Bounded by maxSteps, exactly like generate. If you want raw single-step access with no loop, call provider.generateStream(...) directly.

3. Tool calling #

Declare a tool (name, description, JSON-schema parameters, and the function to run). The SDK orchestrates the whole round-trip automatically.

final weather = Tool(
  name: 'getWeather',
  description: 'Returns the current weather for a city',
  parameters: {
    'type': 'object',
    'properties': {'city': {'type': 'string'}},
    'required': ['city'],
  },
  run: (args) async => '29 °C and humid in ${args['city']}',
);

final response = await client.generate(
  [Message.user('What is the weather in Douala?')],
  tools: [weather],
);
print(response.text); // "It's 29 °C and humid in Douala."

The SDK loops automatically (bounded by maxSteps): the AI asks → getWeather runs → the result is sent back to the AI → final answer. You can register multiple tools; the model picks which to call (and may call several).

The same loop runs on the streaming path — see Streaming with tools.

When a tool fails

By default a tool that throws does not abort the run: the error is handed back to the model as a tool result (flagged isError, which Claude carries on the wire as is_error), so it can retry differently, pick another tool, or tell the user the information is unavailable. The turn you already paid for isn't lost. The same applies when the model asks for a tool you didn't register.

Because a swallowed exception is a silent failure, pass onToolError to keep a trace — it fires in both policies:

final client = LlmClient(
  provider,
  onToolError: (toolName, error, stack) =>
      log.warning('tool $toolName failed', error, stack),
);

Want the old behaviour — the exception reaching you and the run aborting?

final client = LlmClient(provider, toolErrors: ToolErrorPolicy.throwToCaller);

The original exception is rethrown with its stack trace intact.

Parallel tool calls

When the model requests several tools in one turn, they run concurrently — providers emit parallel calls on purpose, and running them in series just adds up the latencies. Results are always re-injected in call order. If your tools share state that doesn't tolerate concurrency:

final client = LlmClient(provider, parallelTools: false);

Controlling tool use

toolChoice constrains how the model arbitrates tools. All three providers support the same four cases, each in its own dialect:

// Answer in prose, even though tools are registered.
await client.generateText(messages, tools: tools, toolChoice: ToolChoice.none);

// Force a tool on the first turn, then let the model answer.
await client.generateText(messages, tools: tools, toolChoice: ToolChoice.any);

toolChoice applies to the first step only. The tool loop then lets the provider arbitrate normally. Without that, any or a named tool would force a tool call on every turn, so the loop would never reach a final answer and would run straight into maxSteps.

Forced tool use (any and named) isn't accepted by every model — Claude Fable 5.1 returns a 400 — while auto and none work everywhere. This is also what limits structured outputs on those models, since they are built on a forced tool.

4. Structured outputs #

Make the model fill a typed "form" instead of returning free-form text. You provide a JSON schema and a fromJson constructor; the model's tool arguments are the object.

class Invoice {
  final String client;
  final double amount;
  Invoice(this.client, this.amount);
  factory Invoice.fromJson(Map<String, dynamic> j) =>
      Invoice(j['client'] as String, (j['amount'] as num).toDouble());
}

final invoice = await client.generateObject<Invoice>(
  [Message.user('Invoice for Metchera, 1,250 EUR incl. tax.')],
  schema: {
    'type': 'object',
    'properties': {
      'client': {'type': 'string'},
      'amount': {'type': 'number'},
    },
    'required': ['client', 'amount'],
  },
  fromJson: Invoice.fromJson,
);
print(invoice.client);  // "Metchera"
print(invoice.amount);  // 1250.0

No runtime reflection in Flutter: the JSON schema and the fromJson are manual in v1. Codegen via annotations is planned for later.

Sampling options #

Tune temperature, topP and stopSequences with a single GenerationOptions, accepted by every LlmClient method. Each provider maps it to its own dialect, so the same object works everywhere. Any field left null is omitted — the model's default applies.

final answer = await client.generateText(
  [Message.user('Write a haiku about Dart.')],
  options: const GenerationOptions(
    temperature: 0.9,
    topP: 0.95,
    stopSequences: ['\n\n'],
  ),
);

maxTokens belongs here too, so the cap can change per request — the provider's own maxTokens stays the default:

await client.generateText(
  [Message.user('One word, please.')],
  options: const GenerationOptions(maxTokens: 16),
);

It works the same on generate, streamText, streamEvents and generateObject.

Escape hatches #

No wrapper models everything, and a missing field shouldn't mean forking the package. Three doors out, all opt-in and all deliberately not portable — they speak each provider's own dialect:

final answer = await client.generateText(
  messages,
  options: const GenerationOptions(
    temperature: 0.5,
    providerOptions: {'generationConfig': {'seed': 42}}, // Gemini dialect
  ),
);

providerOptions is deep-merged into the body: the example above adds seed to the generationConfig the SDK built, rather than replacing it. At an equal key with a non-map value, your value wins — including over the SDK's own. That is the point of a door out, and also its risk.

extraHeaders is applied last, so it can override the SDK's headers. That's what makes it possible to route through your own backend instead of shipping an API key inside a published app:

final provider = OpenAIProvider(
  baseUrl: 'https://my-backend.example/llm',   // your proxy holds the real key
  extraHeaders: {'authorization': 'Bearer $userSessionToken'},
);

raw carries the provider's decoded JSON body. It is null on the streaming path — a response assembled from dozens of SSE chunks has no single body to expose.

Retries & timeouts #

Every provider is created with a RetryPolicy. By default the generate path retries transient failures — HTTP 408/429/5xx, timeouts and dropped connections — with jittered exponential backoff, and every request is bounded by a timeout. Streaming applies the connection timeout only (replaying a started stream is unsafe).

When a rate-limited response carries a Retry-After header, that wins over the backoff: only the provider knows when its window reopens. Both header forms are read — a number of seconds, or an HTTP date. If the server asks for longer than maxRetryAfter, the SDK stops retrying and hands the response back, rather than blocking your request for minutes.

The backoff is jittered: each delay is drawn uniformly from [nominal × (1 - jitter), nominal]. Without it, every client that hits the same 429 retries on the same millisecond and re-saturates the service.

final provider = ClaudeProvider(
  apiKey: myKey,
  retry: const RetryPolicy(
    maxRetries: 3,                            // extra attempts after the first
    initialDelay: Duration(milliseconds: 500),
    backoffFactor: 2.0,                       // 0.5s, 1s, 2s, ... before jitter
    timeout: Duration(seconds: 30),
    maxRetryAfter: Duration(seconds: 30),     // cap on a server-dictated wait
  ),
);

// Opt out entirely:
final noRetry = OpenAIProvider(apiKey: myKey, retry: RetryPolicy.none);

RetryPolicy.parseRetryAfter(header) is public if you need to read the header yourself — it returns null for a missing or unparseable value.

Error handling #

When a provider returns a non-200 HTTP status, or a response cannot be parsed, the SDK throws an LlmException:

try {
  final answer = await client.generateText([Message.user('Hello')]);
  print(answer);
} on LlmException catch (e) {
  print('Provider error ${e.statusCode}: ${e.body}');
}

A StateError is thrown if the tool loop hits maxSteps without a final answer, or if the model requests a tool you didn't register.

Streaming errors #

A stream always ends with either a StreamDone or an error — never quietly. Two subclasses of LlmException cover what can go wrong mid-stream, so a single on LlmException catch still catches everything:

try {
  await for (final chunk in client.streamText([Message.user('Tell me a joke')])) {
    stdout.write(chunk);
  }
} on LlmStreamInterruptedException catch (e) {
  // Keep what arrived, tell the user it was cut short, offer a retry.
  print('\n[truncated after ${e.partial.text.length} chars]');
} on LlmStreamErrorException catch (e) {
  print('\n[provider failed mid-answer: ${e.body}]');
}

Fragments already yielded stay valid in both cases — they are simply incomplete. Terminal markers per provider: message_stop (Claude), [DONE] (OpenAI), and the last chunk's finishReason (Gemini, which has no dedicated end marker).

Resource cleanup #

Each provider owns an internal http.Client. If you create a provider manually (rather than injecting one), close it when you're done:

final provider = ClaudeProvider(apiKey: myKey);
// ... use it ...
provider.close();

For long-lived apps you usually create the provider once and keep it for the process lifetime — no need to close per request.

API reference #

LlmClient #

Every method above also takes an optional GenerationOptions options (see Sampling options).

Core types #

Architecture #

The contract boils down to two methods (generate, generateStream) that each provider implements. All usage logic — the tool loop, streamText, generateObject — is built once in LlmClient, on top of the contract. Providers stay thin: they only translate to/from their own dialect.

LlmClient  ── tool loop (Future *and* streamed), streamText, generateObject
   │
   └── LlmProvider (contract: generate + generateStream)
         ├── ClaudeProvider   ✅ (text, tools, structured outputs, SSE streaming)
         ├── OpenAIProvider   ✅ (same, Chat Completions dialect + local endpoints)
         └── GeminiProvider   ✅ (same, generateContent dialect)

Status & limitations #

Current version: 0.7.0

• ✅ Provider-agnostic core: types, contract, LlmClient (tool loop, generateObject, streamText).
• ✅ All 3 adapters (Claude, OpenAI, Gemini) complete: generate, tool calling, structured outputs (via forced tool), SSE streaming.
• ✅ Local models via the OpenAI adapter (overridable baseUrl, optional key): Ollama, LM Studio, llama.cpp, vLLM.
• ✅ Sampling options (temperature, topP, stopSequences) across all providers.
• ✅ Retries + timeouts with exponential backoff (RetryPolicy) on the generate path.
• ✅ No silent streaming failures: a stream always ends with a StreamDone or an error (LlmStreamErrorException / LlmStreamInterruptedException).
• ✅ Streaming + tools together: the automatic tool loop runs on the streaming path too.
• ✅ Resilient tool loop: a failing tool is handed back to the model instead of aborting the run, tools in one turn run concurrently, and usage is summed across every step on both paths.
• ✅ Rate-limit aware retries: Retry-After (seconds or HTTP date) is honoured over the backoff, which is itself jittered.
• ✅ Full tool arbitration: ToolChoice (auto / none / any / named) across all three dialects, applied to the first step of the loop.
• ✅ Escape hatches: per-request maxTokens, providerOptions merged into the body, extraHeaders, and LlmResponse.raw.
• ✅ 112 tests (mocked client logic + round-trip/SSE for all 3 providers + options/retry + streaming error/interruption + streamed tool loop end-to-end through each provider's dialect + tool-error / usage / concurrency + Retry-After / jitter + tool-choice mapping / escape-hatch coverage).
• 🎯 All 4 building blocks work across the 3 providers with no change to the core — the abstraction is the product, and it held.
• ⬜ Out of scope for v1: embeddings, vision/audio, cost tracking, caching, multi-step agents beyond the tool loop.

Switching on sealed types #

Part and LlmStreamEvent are sealed so that adapters are forced to handle every variant — they encode to the wire, nothing may slip through. The trade-off is on your side: new variants will appear (image, audio, reasoning), and an exhaustive switch over today's variants would then fail to compile.

You pick which behaviour you want, and the two are mutually exclusive:

Forward-compatible — match only the variants you act on and close with _. Nothing breaks on upgrade:

switch (event) {
  case TextDelta(:final text):      stdout.write(text);
  case StreamDone(:final response): print(response.usage);
  case _:                           break; // ToolCallDelta + future variants
}

Told about new variants — enumerate them all and omit _. A new variant then becomes a compile error pointing at the switch, which is what you want when every kind of content must be handled deliberately:

switch (event) {
  case TextDelta(:final text):      stdout.write(text);
  case ToolCallDelta(:final call):  print('tool: ${call.name}');
  case StreamDone(:final response): print(response.usage);
}

Don't combine the two: covering every variant and adding _ makes the analyzer flag the _ as unreachable_switch_case.

LlmResponse.text and LlmResponse.toolCalls don't switch at all, and are the better choice when you only need those.

Migrating from 0.6.x #

The only breaking change is on the LlmProvider contract, so code that uses LlmClient needs no change. Update your own adapters if you wrote any:

   Future<LlmResponse> generate(
     List<Message> messages, {
     List<Tool> tools = const [],
-    String? forceTool,
+    ToolChoice? toolChoice,
     GenerationOptions? options,
   });

   Stream<LlmStreamEvent> generateStream(
     List<Message> messages, {
     List<Tool> tools = const [],
+    ToolChoice? toolChoice,
     GenerationOptions? options,
   });

forceTool: 'x' becomes toolChoice: ToolChoice.named('x'), and the three other cases are newly expressible.

Known limitations #

• On OpenAI, maxTokens is sent as max_tokens; current models expect max_completion_tokens, and the reasoning tiers reject temperature/topP.
• The same Tool.parameters isn't accepted everywhere: Gemini takes only an OpenAPI subset, OpenAI strict mode has its own rules. No schema sanitiser yet.
• Prompt caching is neither settable nor measurable (Usage has no cache counters) — the biggest cost lever still on the table.
• generateObject doesn't validate against your schema or retry a bad parse.
• On Flutter Web, package:http's browser client buffers the whole response, so streamText arrives in one chunk instead of streaming.
• No embeddings, no vision/audio parts, no request cancellation.

Testing #

dart pub get
dart test

License #

See LICENSE.