llm_sdk

The unified toolkit for talking to any LLM from a Dart / Flutter app. One interface, several interchangeable brains.

On the web (JS) there is a polished toolkit for talking to LLMs (think Vercel AI SDK). On the Dart / Flutter side, there was nothing equivalent — everyone re-plumbs their own HTTP calls. llm_sdk is that clean bridge: multi-provider, streaming, tool calling and structured outputs behind one API.

Switching AI = changing one line

Streaming, word by word

---

Table of contents

• Why llm_sdk
• Install
• Quick start
• Providers & configuration
• Local models (Ollama, LM Studio, llama.cpp, vLLM)
• The 4 building blocks
  • Text generation
  • Streaming
  • Tool calling
  • Structured outputs
• Sampling options
• Retries & timeouts
• Error handling
• Resource cleanup
• API reference
• Architecture
• Status & limitations
• Testing
• License

---

Why llm_sdk

Building block	What it does
Multi-provider	The same "buttons" no matter which vendor is behind them.
Streaming	Show the answer word by word, live.
Tool calling	The AI asks to run functions; the SDK orchestrates the round-trip.
Structured outputs	The AI fills a typed Dart object instead of returning free-form text.

The contract a provider must implement is just two methods (generate, generateStream). All the usage logic — the tool loop, streamText, generateObject — is built once in LlmClient, on top of that contract. Switching from Claude to OpenAI to Gemini changes a single line; nothing else in your code moves.

Install

Add the dependency:

dart pub add llm_sdk

or, in a Flutter project:

flutter pub add llm_sdk

or add it manually to your pubspec.yaml:

dependencies:
  llm_sdk: ^0.6.0

Then import it:

import 'package:llm_sdk/llm_sdk.dart';

The only runtime dependency is http, so the package works anywhere Dart runs (CLI, server, Flutter mobile/desktop/web). Requires Dart 3.4 or newer.

Quick start

import 'dart:io';
import 'package:llm_sdk/llm_sdk.dart';

Future<void> main() async {
  // Pick the brain. Switching AI = changing this one line.
  final client = LlmClient(
    ClaudeProvider(apiKey: Platform.environment['ANTHROPIC_API_KEY']!),
  );

  final answer = await client.generateText([
    Message.system('You are a concise assistant.'),
    Message.user('Give me one productivity tip.'),
  ]);

  print(answer);
}

Tip: never hard-code API keys. Read them from environment variables (Platform.environment['...']) or your app's secret storage.

Providers & configuration

Every provider implements the same LlmProvider contract, so they are fully interchangeable inside an LlmClient. They differ only in their constructor options and default model.

Claude (Anthropic)

final provider = ClaudeProvider(
  apiKey: 'sk-ant-...',          // required
  model: 'claude-opus-5',        // default
  maxTokens: 1024,               // default — required by Anthropic
);

OpenAI (and OpenAI-compatible servers)

final provider = OpenAIProvider(
  apiKey: 'sk-...',                       // optional — empty for local servers
  model: 'gpt-5.6-terra',                 // default
  maxTokens: null,                        // optional — null lets the model decide
  baseUrl: 'https://api.openai.com/v1',   // default — override for local models
);

Gemini (Google)

final provider = GeminiProvider(
  apiKey: 'AIza...',                                            // required
  model: 'gemini-3.8-flash',                                    // default
  maxTokens: null,                                              // optional
  baseUrl: 'https://generativelanguage.googleapis.com/v1beta', // default
);

Common options

Option	Type	Notes
apiKey	String	Required for Claude & Gemini. Optional for OpenAI (empty for local servers).
model	String	Model id. The default tracks each provider's current general-purpose model and may change in a minor release — pin it explicitly in production.
maxTokens	int / int?	Required & defaults to 1024 on Claude; optional (null) on OpenAI & Gemini.
baseUrl	String	Available on OpenAI & Gemini to point at a different endpoint.
retry	RetryPolicy	Backoff retries + per-request timeout. See Retries & timeouts.
httpClient	http.Client?	Inject your own client (timeouts, proxy, tests).

Swapping providers

Because the surface is identical, swapping is a one-line change:

final client = LlmClient(ClaudeProvider(apiKey: myKey));
// ... or, without touching any other line:
final client = LlmClient(OpenAIProvider(apiKey: myKey));
final client = LlmClient(GeminiProvider(apiKey: myKey));

You can also tune the tool loop bound:

final client = LlmClient(provider, maxSteps: 8); // default is 5

Local models (Ollama, LM Studio, llama.cpp, vLLM)

Any server that exposes an OpenAI-compatible API works out of the box: just point baseUrl at your local endpoint. The API key is optional — a local server ignores it.

final client = LlmClient(OpenAIProvider(
  baseUrl: 'http://localhost:11434/v1', // Ollama
  model: 'llama3.2',
));

All 4 building blocks (text, streaming, tool calling, structured outputs) stay identical — only the baseUrl changes. No data leaves the machine.

Common endpoints:

Server	baseUrl
Ollama	http://localhost:11434/v1
LM Studio	http://localhost:1234/v1
llama.cpp / vLLM	their own endpoint

The 4 building blocks

1. Text generation

generate returns a full LlmResponse (text, usage, finish reason, tool calls). generateText is the shortcut that returns the final string directly.

final response = await client.generate([
  Message.user('Summarize relativity in one sentence.'),
]);

print(response.text);                       // the answer
print(response.usage?.totalTokens);         // token count, if provided
print(response.finishReason);               // FinishReason.stop, length, ...

Build conversations by stacking messages:

final messages = [
  Message.system('You are a helpful translator.'),
  Message.user('Translate "good morning" to French.'),
  Message.assistant('Bonjour.'),
  Message.user('And to Spanish?'),
];
final reply = await client.generateText(messages);

2. Streaming

Stream the answer word by word for a typewriter effect:

await for (final chunk in client.streamText([Message.user('Tell me a joke')])) {
  stdout.write(chunk); // typewriter effect
}

Need the raw event stream (text deltas, tool calls, and the assembled final response)? Use streamEvents:

await for (final event in client.streamEvents([Message.user('Hi')])) {
  switch (event) {
    case TextDelta(:final text):       stdout.write(text);
    case ToolCallDelta(:final call):   print('tool: ${call.name}');
    case StreamDone(:final response):  print('\nusage: ${response.usage}');
  }
}

Streaming with tools

Both streaming methods take tools and run the same automatic tool loop as generate. The model can narrate, ask for a tool, and keep streaming its final answer — all inside one stream:

await for (final chunk in client.streamText(
  [Message.user('Compare the weather in Douala and Yaoundé.')],
  tools: [weather], // ← runs automatically, mid-stream
)) {
  stdout.write(chunk);
}

TextDelta and ToolCallDelta are forwarded for every step, so a UI can show "running getWeather…" as it happens. StreamDone is emitted once, at the very end — never per step — and its usage is the sum across all steps, so a tool round-trip doesn't go uncounted (generate sums the same way).

Bounded by maxSteps, exactly like generate. If you want raw single-step access with no loop, call provider.generateStream(...) directly.

3. Tool calling

Declare a tool (name, description, JSON-schema parameters, and the function to run). The SDK orchestrates the whole round-trip automatically.

final weather = Tool(
  name: 'getWeather',
  description: 'Returns the current weather for a city',
  parameters: {
    'type': 'object',
    'properties': {'city': {'type': 'string'}},
    'required': ['city'],
  },
  run: (args) async => '29 °C and humid in ${args['city']}',
);

final response = await client.generate(
  [Message.user('What is the weather in Douala?')],
  tools: [weather],
);
print(response.text); // "It's 29 °C and humid in Douala."

The SDK loops automatically (bounded by maxSteps): the AI asks → getWeather runs → the result is sent back to the AI → final answer. You can register multiple tools; the model picks which to call (and may call several).

The same loop runs on the streaming path — see Streaming with tools.

When a tool fails

By default a tool that throws does not abort the run: the error is handed back to the model as a tool result (flagged isError, which Claude carries on the wire as is_error), so it can retry differently, pick another tool, or tell the user the information is unavailable. The turn you already paid for isn't lost. The same applies when the model asks for a tool you didn't register.

Because a swallowed exception is a silent failure, pass onToolError to keep a trace — it fires in both policies:

final client = LlmClient(
  provider,
  onToolError: (toolName, error, stack) =>
      log.warning('tool $toolName failed', error, stack),
);

Want the old behaviour — the exception reaching you and the run aborting?

final client = LlmClient(provider, toolErrors: ToolErrorPolicy.throwToCaller);

The original exception is rethrown with its stack trace intact.

Parallel tool calls

When the model requests several tools in one turn, they run concurrently — providers emit parallel calls on purpose, and running them in series just adds up the latencies. Results are always re-injected in call order. If your tools share state that doesn't tolerate concurrency:

final client = LlmClient(provider, parallelTools: false);

4. Structured outputs

Make the model fill a typed "form" instead of returning free-form text. You provide a JSON schema and a fromJson constructor; the model's tool arguments are the object.

class Invoice {
  final String client;
  final double amount;
  Invoice(this.client, this.amount);
  factory Invoice.fromJson(Map<String, dynamic> j) =>
      Invoice(j['client'] as String, (j['amount'] as num).toDouble());
}

final invoice = await client.generateObject<Invoice>(
  [Message.user('Invoice for Metchera, 1,250 EUR incl. tax.')],
  schema: {
    'type': 'object',
    'properties': {
      'client': {'type': 'string'},
      'amount': {'type': 'number'},
    },
    'required': ['client', 'amount'],
  },
  fromJson: Invoice.fromJson,
);
print(invoice.client);  // "Metchera"
print(invoice.amount);  // 1250.0

No runtime reflection in Flutter: the JSON schema and the fromJson are manual in v1. Codegen via annotations is planned for later.

Sampling options

Tune temperature, topP and stopSequences with a single GenerationOptions, accepted by every LlmClient method. Each provider maps it to its own dialect, so the same object works everywhere. Any field left null is omitted — the model's default applies.

final answer = await client.generateText(
  [Message.user('Write a haiku about Dart.')],
  options: const GenerationOptions(
    temperature: 0.9,
    topP: 0.95,
    stopSequences: ['\n\n'],
  ),
);

It works the same on generate, streamText, streamEvents and generateObject.

Retries & timeouts

Every provider is created with a RetryPolicy. By default the generate path retries transient failures — HTTP 408/429/5xx, timeouts and dropped connections — with jittered exponential backoff, and every request is bounded by a timeout. Streaming applies the connection timeout only (replaying a started stream is unsafe).

When a rate-limited response carries a Retry-After header, that wins over the backoff: only the provider knows when its window reopens. Both header forms are read — a number of seconds, or an HTTP date. If the server asks for longer than maxRetryAfter, the SDK stops retrying and hands the response back, rather than blocking your request for minutes.

The backoff is jittered: each delay is drawn uniformly from [nominal × (1 - jitter), nominal]. Without it, every client that hits the same 429 retries on the same millisecond and re-saturates the service.

final provider = ClaudeProvider(
  apiKey: myKey,
  retry: const RetryPolicy(
    maxRetries: 3,                            // extra attempts after the first
    initialDelay: Duration(milliseconds: 500),
    backoffFactor: 2.0,                       // 0.5s, 1s, 2s, ... before jitter
    timeout: Duration(seconds: 30),
    maxRetryAfter: Duration(seconds: 30),     // cap on a server-dictated wait
  ),
);

// Opt out entirely:
final noRetry = OpenAIProvider(apiKey: myKey, retry: RetryPolicy.none);

Option	Default	Meaning
maxRetries	2	Extra attempts after the first (0 disables).
initialDelay	400 ms	Nominal delay before the first retry.
backoffFactor	2.0	Multiplier applied between attempts.
jitter	0.5	Random spread, 0.0–1.0. 0 gives a strictly deterministic backoff.
timeout	60 s	Per-request deadline before a TimeoutException.
retryStatusCodes	{408, 429, 500, 502, 503, 504}	Statuses treated as transient.
respectRetryAfter	true	Honour a Retry-After header over the backoff.
maxRetryAfter	60 s	Longest server-dictated wait accepted; beyond it, stop retrying.
random	null	Inject a seeded Random to make the jitter deterministic in tests.

RetryPolicy.parseRetryAfter(header) is public if you need to read the header yourself — it returns null for a missing or unparseable value.

Error handling

When a provider returns a non-200 HTTP status, or a response cannot be parsed, the SDK throws an LlmException:

try {
  final answer = await client.generateText([Message.user('Hello')]);
  print(answer);
} on LlmException catch (e) {
  print('Provider error ${e.statusCode}: ${e.body}');
}

Field	Type	Meaning
statusCode	int	HTTP status from the provider (0 if the error isn't network-level).
body	String	Raw response body, or an error message.

A StateError is thrown if the tool loop hits maxSteps without a final answer, or if the model requests a tool you didn't register.

Streaming errors

A stream always ends with either a StreamDone or an error — never quietly. Two subclasses of LlmException cover what can go wrong mid-stream, so a single on LlmException catch still catches everything:

Exception	When	Extra
LlmStreamErrorException	The provider announced an error during generation (HTTP was already 200): overload, rate limit, safety stop.	body holds the provider's raw payload.
LlmStreamInterruptedException	The stream ended before its terminal marker — dropped connection, server closing early.	partial holds the response assembled so far.

try {
  await for (final chunk in client.streamText([Message.user('Tell me a joke')])) {
    stdout.write(chunk);
  }
} on LlmStreamInterruptedException catch (e) {
  // Keep what arrived, tell the user it was cut short, offer a retry.
  print('\n[truncated after ${e.partial.text.length} chars]');
} on LlmStreamErrorException catch (e) {
  print('\n[provider failed mid-answer: ${e.body}]');
}

Fragments already yielded stay valid in both cases — they are simply incomplete. Terminal markers per provider: message_stop (Claude), [DONE] (OpenAI), and the last chunk's finishReason (Gemini, which has no dedicated end marker).

Resource cleanup

Each provider owns an internal http.Client. If you create a provider manually (rather than injecting one), close it when you're done:

final provider = ClaudeProvider(apiKey: myKey);
// ... use it ...
provider.close();

For long-lived apps you usually create the provider once and keep it for the process lifetime — no need to close per request.

API reference

LlmClient

Member	Signature	Description
constructor	LlmClient(LlmProvider provider, {int maxSteps = 5, ToolErrorPolicy toolErrors = ToolErrorPolicy.feedToModel, bool parallelTools = true, ToolErrorCallback? onToolError})	Wraps a provider. maxSteps bounds the tool loop; see When a tool fails and Parallel tool calls.
generate	Future<LlmResponse> generate(List<Message>, {List<Tool> tools})	Full response, with automatic tool loop.
generateText	Future<String> generateText(List<Message>, {List<Tool> tools})	Convenience: returns the final text.
streamText	Stream<String> streamText(List<Message>, {List<Tool> tools})	Text chunks, word by word, with automatic tool loop.
streamEvents	Stream<LlmStreamEvent> streamEvents(List<Message>, {List<Tool> tools})	Typed event stream, with automatic tool loop. One StreamDone at the end.
generateObject	Future<T> generateObject<T>(List<Message>, {required Map schema, required T Function(Map) fromJson, String description})	Typed structured output.

Every method above also takes an optional GenerationOptions options (see Sampling options).

Core types

Type	Purpose
Message	A conversation turn. Factories: Message.system/user/assistant(text).
Part (sealed)	TextPart, ToolCallPart, ToolResultPart.
Tool	A callable tool: name, description, parameters (JSON schema), run.
ToolErrorPolicy	feedToModel (default) or throwToCaller, for a tool that throws.
ToolErrorCallback	void Function(String toolName, Object error, StackTrace) — observe tool failures.
LlmResponse	message, text, toolCalls, usage, finishReason.
Usage	inputTokens, outputTokens, totalTokens.
FinishReason	stop, length, toolUse, contentFilter, unknown.
LlmStreamEvent (sealed)	TextDelta, ToolCallDelta, StreamDone.
LlmException	Thrown on provider/HTTP errors. Base class of the two below.
LlmStreamErrorException	The provider errored mid-stream (HTTP was 200).
LlmStreamInterruptedException	The stream was cut before its end marker; carries partial.

Architecture

The contract boils down to two methods (generate, generateStream) that each provider implements. All usage logic — the tool loop, streamText, generateObject — is built once in LlmClient, on top of the contract. Providers stay thin: they only translate to/from their own dialect.

LlmClient  ── tool loop (Future *and* streamed), streamText, generateObject
   │
   └── LlmProvider (contract: generate + generateStream)
         ├── ClaudeProvider   ✅ (text, tools, structured outputs, SSE streaming)
         ├── OpenAIProvider   ✅ (same, Chat Completions dialect + local endpoints)
         └── GeminiProvider   ✅ (same, generateContent dialect)

Status & limitations

Current version: 0.6.0

• ✅ Provider-agnostic core: types, contract, LlmClient (tool loop, generateObject, streamText).
• ✅ All 3 adapters (Claude, OpenAI, Gemini) complete: generate, tool calling, structured outputs (via forced tool), SSE streaming.
• ✅ Local models via the OpenAI adapter (overridable baseUrl, optional key): Ollama, LM Studio, llama.cpp, vLLM.
• ✅ Sampling options (temperature, topP, stopSequences) across all providers.
• ✅ Retries + timeouts with exponential backoff (RetryPolicy) on the generate path.
• ✅ No silent streaming failures: a stream always ends with a StreamDone or an error (LlmStreamErrorException / LlmStreamInterruptedException).
• ✅ Streaming + tools together: the automatic tool loop runs on the streaming path too.
• ✅ Resilient tool loop: a failing tool is handed back to the model instead of aborting the run, tools in one turn run concurrently, and usage is summed across every step on both paths.
• ✅ Rate-limit aware retries: Retry-After (seconds or HTTP date) is honoured over the backoff, which is itself jittered.
• ✅ 91 tests (mocked client logic + round-trip/SSE for all 3 providers + options/retry + streaming error/interruption + streamed tool loop end-to-end through each provider's dialect + tool-error / usage / concurrency + Retry-After / jitter coverage).
• 🎯 All 4 building blocks work across the 3 providers with no change to the core — the abstraction is the product, and it held.
• ⬜ Out of scope for v1: embeddings, vision/audio, cost tracking, caching, multi-step agents beyond the tool loop.

Known limitations

• maxTokens is fixed per provider instance; it can't be set per request.
• No escape hatch for provider-specific fields: LlmResponse has no raw, and GenerationOptions covers only temperature / topP / stopSequences.

Testing

dart pub get
dart test

License

See LICENSE.