flutter_cockpit_test

Write normal Flutter integration tests with Cockpit's real locator, control, evidence, and diagnostics engine.

English · 简体中文

flutter_cockpit_test is a development-only test facade. It keeps Flutter's official integration_test runner and adds the parts that flutter_test cannot provide by itself: Cockpit's source-friendly Element selectors, real hit-tested actions, lazy-list reveal, compact snapshots, native screenshots, recording, viewport control, and explicit host/system actions.

Install #

Add it to the application's development dependencies, an optional development module, or another test-only package; never import it from production application code:

flutter pub add --dev flutter_cockpit_test

The package is intended for a direct cockpit/ entrypoint or non-published Cockpit module that already uses flutter_cockpit. It does not depend on the Cockpit CLI, daemon, MCP server, or any secret store.

Quick start #

import 'package:flutter/gestures.dart';
import 'package:flutter/material.dart';
import 'package:flutter_cockpit_test/flutter_cockpit_test.dart';

void main() {
  cockpitTestWidgets(
    'creates a task',
    app: buildDevelopmentApp,
    body: (cockpit) async {
      await cockpit.tap('New task');
      await cockpit.type('Buy milk', into: 'Task title');
      await cockpit.tap('Save');
      await cockpit.expectText('Task created', 'Task created');
    },
  );
}

Widget buildDevelopmentApp() {
  return const MaterialApp(home: TaskEditorScreen());
}

The helper wraps a plain Flutter widget in FlutterCockpitApp. If the builder already returns FlutterCockpitApp, it is mounted as-is and its existing Cockpit root is reused. This makes migration from an existing development shell incremental.

Selectors use the same syntax as cockpit dev:

await cockpit.tap('#save');
await cockpit.hover('More options');
await cockpit.tap(null, at: const Offset(400, 300),
    device: PointerDeviceKind.mouse, buttons: kSecondaryButton);
await cockpit.tap('Dialog >> FilledButton["Continue"]');
await cockpit.type('hello', into: '@message');
await cockpit.scroll('Settings >> Text["Advanced"]', align: 'center');
await cockpit.wheel(
  target: '#list',
  delta: const Offset(0, 120),
  steps: 2,
);
await cockpit.dragTo(
  from: 'Drag third',
  to: 'Drop first',
  placement: 'before',
);

Plain text is exact. Use #id, @key, widget type, ancestor chains, and multiple conditions when source context gives you a stronger locator. No business Key or Semantics changes are required for Cockpit's Element plane.

The facade covers the complete Flutter interaction loop directly: pointer gestures (tap, hover, longPress, doubleTap, drag, dragTo, fling, swipe, pinch, rotate, panZoom, multiTouch, wheel), text and keyboard input (type, clear, copy, paste, focus, setTextEditingValue, selectText, keyDown, keyUp, hotkey, press), controls and navigation (increase, decrease, showOnScreen, scroll, waitFor, waitForUi, waitForRoute, back, dismiss, dismissKeyboard), assertions and evidence (expectVisible, expectText, screenshot, snapshot, watch, execute). The facade also provides command-recorded collectSnapshot() and clearNetworkActivity(); host-only expectScreenshot() and travel() are available when an explicit hostCommand adapter is configured. Each command advances Flutter's test clock through the same commit and reveal logic used by the live bridge, so route pushes and async UI updates do not need hand-written sleeps. Use cockpit.flutter when a test intentionally needs a Flutter-only matcher or custom pump.

All gestures are real hit-tested pointer events. Coordinate input is available as at when a target is not discoverable; device and buttons cover mouse, stylus, and touch-sensitive behavior without changing the app. wheel sends real PointerScrollEvent signals to Scrollable, custom Listener(onPointerSignal: ...), and trackpad-aware widgets. Its delta is applied per event; use steps, interval, device, or at only when the scenario needs them. dragTo resolves both targets and computes a drop point for reorderable lists, kanban boards, and drop zones; choose before|center|after only when placement matters.

Every facade command has a 10-second default timeout. Override one known-slow call with timeout; the value must be positive and no longer than one hour:

await cockpit.waitForRoute('/reports', timeout: const Duration(seconds: 30));
await cockpit.tap('Refresh', timeout: const Duration(seconds: 5));

CockpitTestOptions.commandTimeout changes the default for all in-app commands. Native capture, recording, viewport, and capability calls use a separate two-minute default through nativeTimeout, and each native method also accepts its own timeout. A timed-out recording start requests cancellation before the timeout is reported.

Native and host capabilities #

Flutter's test binding controls Flutter widgets. Cockpit's native facade covers the app-window capabilities exposed by the installed plugin:

final available = await cockpit.native.queryCaptureAvailability();
if (available) {
  final capture = await cockpit.native.captureScreenshot(
    name: 'task-created',
    timeout: const Duration(seconds: 30),
  );
  // capture.screenshot.artifact.relativePath identifies the evidence artifact.
}

final recording = await cockpit.native.queryRecordingCapabilities();
if (recording.supportsNativeRecording) {
  await cockpit.native.startRecording(
    name: 'task-flow',
    timeout: const Duration(minutes: 2),
  );
  // exercise the flow
  final result = await cockpit.native.stopRecording(
    timeout: const Duration(seconds: 30),
  );
  // result.artifact or result.sourceFilePath identifies the recording.
}

final resized = await cockpit.native.resizeViewport(width: 800, height: 600);

OS dialogs, app links, accessibility controls, and other host actions belong to Cockpit's system plane. They are intentionally explicit and supplied by the test host:

await cockpit.host.action(
  'openUri',
  parameters: {'uri': 'myapp://tasks/42'},
);

Configure CockpitTestOptions.hostCommand with a host adapter that forwards the command to Cockpit's public control API. Without that callback, host actions fail immediately with a useful configuration error; no external side effect is guessed or hidden.

Flutter APIs remain available #

CockpitTester.flutter is the original WidgetTester. Use it for custom matchers, golden assertions, pump control, or APIs that are intentionally outside Cockpit's command surface. CockpitTester.execute accepts a complete CockpitCommand when a test needs a lower-level operation.

Every executed command is recorded into the in-app Cockpit session and a compact cockpit entry is merged into integration_test's reportData. Large snapshots and binary evidence are kept as artifacts; they are not dumped into test output.

For command-level evidence, use collectSnapshot() when the snapshot itself must be recorded as a step, and clearNetworkActivity() before a flow that needs an isolated HTTP/SSE/WebSocket index. snapshot() remains the synchronous low-overhead read. Host-only capabilities stay explicit: expectScreenshot() delegates baseline comparison to the configured host adapter and travel() replays a validated simulated-location route with CockpitTravelPoint values. Neither host method guesses a native action when hostCommand is absent.

await cockpit.clearNetworkActivity();
final before = await cockpit.collectSnapshot();
await cockpit.tap('#open-chart');
final after = await cockpit.collectSnapshot();
expect(after.routeName, before.routeName);

await cockpit.expectScreenshot(baseline: 'test/baselines/chart.png');
await cockpit.travel(const [
  CockpitTravelPoint(latitude: 31.20, longitude: 121.50),
  CockpitTravelPoint(latitude: 31.21, longitude: 121.51),
]);

VM debugger controls #

When the test runner exposes a Dart VM Service, cockpit.debugger provides the common Dart-Code debugger loop without requiring a separate IDE connection. It uses one lazy, reusable VM session and keeps reads bounded:

final state = await cockpit.debugger.status();
if (state.runnable == true) {
  await cockpit.debugger.pause();
}
final paused = await cockpit.debugger.status(stackLimit: 32);
final value = await cockpit.debugger.evaluateInFrame(0, 'cart.length');
await cockpit.debugger.resume();

The facade also supports stack, evaluate, getObject, source addBreakpoint/removeBreakpoint, setBreakpointEnabled, setPauseMode, setLibraryDebuggable, reloadSources, and an explicit callServiceExtension for Flutter inspector or custom service extensions. The returned models contain verified source locations, frame variables, pause state, and bounded value summaries; large object pages are fetched only when getObject is called with an explicit offset/count. available is a side-effect-free probe, and unsupported web/release runners report an unavailable VM Service through the typed exception rather than fabricating debug data.

Performance profiling #

Profile an interaction with the same test clock and frame pipeline used by the app. Cockpit records raw vsync and raster-finish wall-time timestamps together with engine FrameTiming values (build, raster, vsync, total span, raster-cache usage, jank budget, and p50/p90/p99/worst values). On native Flutter targets it also captures the official integration-test VM timeline and GC events plus bounded process RSS samples; web reports the timeline and memory as unavailable instead of fabricating data. If a local flutter test process does not expose a VM Service URI, the action still runs normally and the report records unavailable:vm; flutter drive --profile --no-dds and native instrumentation runs keep the official VM timeline. flutter test integration_test/... is debug-only, and the non-web Flutter Driver rejects flutter drive --release. Use a native XCTest, Android instrumentation, or device-lab harness for release artifacts; VM-backed timeline, CPU, heap, GC, and DevTools data are unavailable unless that harness provides an equivalent collector:

final report = await cockpit.profile(
  () async {
    await cockpit.tap('#open-list');
    await cockpit.scroll('#list');
  },
  name: 'open-list',
  streams: const <String>['Dart', 'GC', 'Embedder'],
);
expect(report.summary.jankCount, 0);

Use beginPerformance/end when one integration test needs separate performance windows. Windows are sequential and each end returns the real report for that segment; an unclosed window is closed automatically during test teardown:

final openList = await cockpit.beginPerformance(name: 'open-list');
await cockpit.tap('Open list');
final openListReport = await openList.end();

final checkout = await cockpit.beginPerformance(
  name: 'checkout',
  mode: CockpitPerformanceMode.light,
);
await cockpit.tap('Checkout');
final checkoutReport = await checkout.end();

CockpitTestOptions.requiredBuildMode only verifies the runner's actual debug, profile, or release mode. It does not relaunch the app or claim a different mode. flutter test integration_test/... is debug; use flutter drive --profile --no-dds for VM-backed profile evidence. The non-web Flutter driver rejects release mode, so release performance evidence must come from a native release harness.

Native captures also sample process RSS every 100ms by default and retain the start/end/min/max/average/peak/delta summary plus the bounded sample timeline. Set memory: false when the extra process metric is irrelevant; use sampleEvery to trade sampling overhead for temporal resolution, streams and timeline to choose VM tracing, and maxEvents to bound retained timeline events. Unsupported targets leave memory unavailable rather than reporting zero. For a diagnostic capture, trackBuilds, trackUserBuilds, trackLayouts, and trackPaints enable Flutter's real per-widget/per-render-object timeline spans, matching the corresponding DevTools switches. They are off by default because the extra instrumentation changes timings, and Cockpit restores the previous global flags after the capture.

The capture also requests the VM Service getProcessMemoryUsage tree by default (vmMemory: true). This is separate from platform RSS: it retains the VM's before/after process buckets, sizes, top children, and explicit dropped child counts under devtools.vmem. The tree is bounded to keep test memory and exports predictable; set vmMemory: false when a VM memory map is not part of the investigation. Unsupported runtimes report the metric as unavailable.

profile() also samples the VM data behind DevTools' CPU Profiler and Memory views when a VM service is available. CPU stacks and bounded allocation classes are retained in the complete report; the compact test result keeps only sample counts. The same capture also records VM heap samples over time, isolate health snapshots for every discovered isolate, the VM Isolate stream's bounded lifecycle events (start, runnable, update, reload, exit, and extension registration, including the registered RPC when the VM provides it), and timeline recorder stream metadata. New/ended isolates and retention drops are visible in the HTML runtime panel and complete JSON; the compact result keeps only counts. Use cpu: false or heap: false for a frame-only capture, and tune maxCpuSamples, maxHeapClasses, or maxHeapSamples for long scenarios:

CPU functions retain verified VM source locations when the runtime provides them, including resolved URI, line, and column. Heap reports also retain the VM allocation-accumulator reset and last service-GC timestamps, while isolate snapshots include pause-on-exit, exception-pause mode, and root-library URI. Missing VM fields stay absent rather than being inferred.

The capture also listens to the VM Logging and Debug streams by default. Logging records keep the message, severity, logger, error, and stack text only when the VM already supplied them; debug records keep pause/resume/exception/ reload context and verified source locations from the notification. These are bounded metadata records, not implicit object evaluation, so they cannot block the tested action. Set logs: false or debug: false when a capture must exclude one stream; maxLogs and maxDebug tune their bounded retention for very long scenarios. Isolate snapshots also retain new/old generation heap spaces and breakpoint counts when the runtime exposes them.

For a class-specific allocation investigation, first obtain its VM class id from the heap report, then opt in to call-stack tracing for at most 20 classes. This extra VM stream is intentionally disabled by default because it changes allocation profiling overhead:

final report = await cockpit.profile(
  () => runScenario(),
  allocationClassIds: <String>['classes/123'],
);
// report.devTools?.allocationTraces contains only the selected classes.

Set perfetto: true when the exact VM CPU/timeline proto is needed for an offline Perfetto viewer. Regular results keep only bounded metadata; complete HTML/JSON exports retain the base64 payload, and native hosts can write standalone .pftrace files for every capture with exportPerformancePerfetto():

final report = await cockpit.profile(
  () => runScenario(),
  perfetto: true,
);
final tracePaths = await cockpit.exportPerformancePerfetto();

Perfetto is best-effort and recorder/platform dependent. If the VM does not support the RPC or the active recorder writes directly to the OS/file, the capture remains valid and coverage marks that trace unavailable.

Memory or streaming: choose per capture #

Short captures use the normal profile() path and keep their retained report in memory, which makes immediate assertions and the self-contained HTML export simple. For hours-long journeys, explicitly open a JSONL archive and pass it to profile; frames, VM/plugin events, heap samples, RSS samples, logs, and debug records are appended to rotating files while the report remains a small in-memory projection:

final archive = await cockpit.openPerformanceArchive(name: 'overnight-flow');
await cockpit.profile(
  () => runOvernightJourney(),
  name: 'overnight-flow',
  archive: archive,
);
// The test teardown closes registered archives and flushes the manifest.

The explicit archive defaults to lossless, so it does not impose a pending queue limit and preserves every record. If a constrained runner needs a hard queue bound, opt into CockpitPerformanceArchiveMode.low and set maxPendingBytes; drops are counted in the manifest and report. Chunk size, flush interval, polling interval, and queue limit are all configurable. archive.info.manifest is the only path needed to locate the chunk set. The manifest stores chunk names relative to itself, so a downloaded CI artifact can be moved as one directory without rewriting paths; archive.info.chunks still returns absolute local paths for direct file access. exportPerformanceJsonl() provides the same format for already completed reports without constructing one giant JSON array. When Android, iOS, macOS, or CI workers produce separate archives, combine them incrementally with CockpitPerformanceArchive.merge([...]). Inputs may be manifests or individual .jsonl/.jsonl.part chunks; the output preserves source order, validates each record, namespaces duplicate capture ids, and returns a new manifest path. It does not globally sort events from different devices because their monotonic clocks are unrelated. The generated HTML embeds the small in-memory report. When opened directly as file://, browser security prevents it from scanning project paths or fetching external JSON/JSONL automatically; serve the report from a local HTTP server or use an explicit browser file picker for a selected manifest/chunk set. Never embed a multi-gigabyte stream in the HTML itself. The standalone HTML report shows archive size, state, and chunk paths; it keeps large streams out of the page and renders retained summaries responsively.

When the VM exposes an isolate group, the heap report also keeps group-level before/after memory points. This covers multi-isolate apps without sampling every group on every tick.

final report = await cockpit.profile(
  () => runScenario(),
  cpu: true,
  heap: true,
  maxCpuSamples: 20000,
  maxHeapClasses: 100,
);

The same facade exposes DevTools' visual diagnostics without changing the production app:

cockpit.debug.apply(
  paintSize: true,
  repaintRainbow: true,
  performanceOverlay: true,
  timeScale: 5,
);
// The test harness restores every switch at tear-down.

GPU/shader values are evidence-only: Cockpit reports matching VM timeline signals when Flutter emits them and clearly marks platform counters as unavailable otherwise. It never substitutes guessed GPU numbers.

The complete bounded report is stored under cockpit.performance.open-list in IntegrationTestWidgetsFlutterBinding.reportData; the normal Cockpit result contains only the compact summary. dropped counts are explicit when a configured retention bound is reached; aggregates then describe the retained sample only. Empty phases omit duration aggregates rather than reporting a fabricated zero, and fps is omitted when the original engine timestamps cannot establish a strictly increasing cadence. The phase budget is derived from the target display refresh rate when Flutter exposes it, otherwise the report records the exact rounded 60Hz fallback interval (16,667µs). The report also records debug, profile, or release; debug timings are diagnostic and must not be used as release performance evidence. Never treat a missing or unavailable metric as zero.

The HTML report adds source evidence only when VM event arguments contain a file, URL, symbol, or line. Frame timings alone do not identify Dart code, so the report never invents a source location.

Instrumentation plugins and AOP adapters #

Use CockpitPerformancePlugin when the application exposes a development hook for repository, network, database, rendering, or business operations. Extend it for custom plugins. open creates a new CockpitPerformancePluginRun for each capture; put mutable subscriptions and counters on that Run so repeated or concurrent captures stay isolated. The plugin is started only for one explicit capture and does not install a global listener:

The complete runnable version of this pattern is in example/performance_plugin.dart.

final report = await cockpit.profile(
  () => runCheckoutFlow(),
  plugins: <CockpitPerformancePlugin>[
    CheckoutPlugin(),
  ],
);

Use instant for a point event, begin/end or trace for a duration, and counter/sample for numeric values. Events use the same monotonic timestamp axis as VM timeline events and are merged into report.events; each carries the plugin source and optional isolate/source location. Per-plugin limits, sampling, category filters, payload depth/size, and drop/invalid/truncation counts keep captures bounded. A throwing plugin is marked failed without failing the measured action. Inspect report.plugins for attribution and statistics; compact output keeps counts, while full JSON, HTML, and the downloaded Chrome trace retain the bounded event detail. AOP adapters should call the sink from explicit development hooks; implicit production-wide instrumentation is intentionally unsupported.

For a small stateless hook, use CockpitPerformancePlugin.callbacks(...) with setup and optional cleanup callbacks. Use a custom Run whenever the hook owns a subscription, timer, buffer, or other mutable resource.

Plugin setup and cleanup are each bounded by the plugin's CockpitPerformancePluginOptions.lifecycleTimeout (two seconds by default). The measured window closes before plugin cleanup and report generation, so teardown work is not attributed to the profiled interaction.

The DevTools section includes a VM heap trend chart, CPU/heap/GPU summaries, VM identity and isolate inventory, all-isolate lifecycle snapshots/events, and recorder/stream metadata. Isolate snapshots and the VM runtime index include the service-extension RPC names reported by the VM, so a missing DevTools capability can be diagnosed from the capture without guessing. Each section has a compact Details action that opens a native dialog with bounded JSON previews. The full retained arrays are embedded in the HTML report and are also available from the JSON export API, so opening a dialog does not freeze the report. CPU details also include verified sample stack paths aggregated from VM function indexes, without inventing source code.

The report also includes Operation hotspots, aggregating each actual VM event category/name into event count, timed-event count, total duration, p90, and longest span. This answers “what is slow?” before opening the raw timeline. The source column is evidence-only and appears only when the VM event arguments provide a location. The same bounded projection is included in fullJson() under each capture's analysis field; original events remain unchanged. The same analysis records GC event count, timed pause total, p50, p90, and max when the retained timeline contains real GC markers. The HTML cache panel shows these pause metrics alongside new/old collection counts.

Each cockpitTestWidgets run also records cold-start milestones in the compact cockpit.startup entry and in the HTML report: app build/mount, first pumped frame, and initial-ready time. The clock begins immediately before the app builder, so the values are honest Dart-harness measurements. Native process launch time is not inferred when the host cannot provide it.

Host-side integration_test_driver.dart files should import package:flutter_cockpit_test/flutter_cockpit_test_report.dart; this pure-Dart entrypoint exports the report models and HTML renderer without loading dart:ui.

Open a complete offline HTML report #

CockpitTester.exportPerformanceHtml() writes one self-contained file for the captures completed by the current test. It includes a report switcher, frame pacing and budget chart, VM timeline lanes, phase percentiles, cache/GC pressure, searchable event arguments, paged frame/event tables, and the exact raw JSON payload. It works without a server or external assets:

final htmlPath = await cockpit.exportPerformanceHtml(
  title: 'Task flow performance',
  // path: 'build/reports/task-flow.html', // optional
);
// Pass htmlPath to a human or CI artifact collector.

For a machine-readable artifact, use performanceJson() or exportPerformanceJson(). This is the full canonical bundle for every completed capture, not the compact integration_test result:

final jsonPath = await cockpit.exportPerformanceJson(
  title: 'Task flow performance',
  // path: 'build/reports/task-flow.json', // optional
);

The exported JSON conforms to the published cockpit.performance/v2 schema, cockpit_protocol/schema/cockpit.performance.v2.schema.json. The schema also validates the single-capture JSON stored under cockpit.performance.<name> in IntegrationTestWidgetsFlutterBinding.reportData, so downstream consumers can validate either surface with one document.

Both export methods preserve all retained frames, VM events and arguments, memory samples, VM heap samples, allocation classes, isolate snapshots, timeline stream lists, startup milestones, and explicit retention/drop counts. The terminal/report output remains compact; exports retain the complete recorded detail. The only limits are the capture retention settings (maxEvents, frame retention, and memory sampling), and those limits are recorded in the export.

The default path is a unique file under build/cockpit/performance/. For a custom host, CockpitPerformanceHtml.render(report) or CockpitPerformanceHtml.renderMany(reports) returns the HTML string without touching the file system. renderMany keeps each capture's real platform and build mode in the selector and comparison table, so Android/iOS/macOS/Web results can share one view without collapsing unavailable metrics into zeros. CockpitPerformanceHtml.fullJson(reports) returns the same complete canonical bundle as a JSON string. JSON remains the canonical machine-readable export; the HTML is the human-facing view with relative-time hover charts, jank distribution, frame cadence, raster-cache trend, VM category cost, a jank/stall evidence table, operation hotspots, separate memory/cache/GC views, and a duration-based VM flame view when spans are available. When startup data is supplied it also renders the app-build, first-frame, and ready milestones as a chart.

The report includes a DevTools coverage panel so unavailable data is obvious: FrameTiming, raster cache, VM timeline, GC, process RSS, and harness cold-start milestones are marked only when the capture actually contains them. CPU sampling, heap/allocation profiling, and matching GPU/shader timeline signals are marked only when the VM actually returns them; unsupported platform counters remain unavailable rather than being fabricated. Use Cockpit's network evidence for HTTP/SSE/WebSocket traffic. The top-bar Download timeline action exports the retained VM events as a Chrome trace-compatible traceEvents JSON file for timeline viewers; the complete FrameTiming and memory data remains in the report JSON and HTML charts. You can also generate that file from a host driver with CockpitPerformanceHtml.timelineJson(report). Compact reports do not retain async/flow event IDs, so those phases are lowered to self-contained instant or duration events to keep the exported trace importable.

This intentionally mirrors the evidence that can be collected without attaching an interactive DevTools session:

The normal test output is intentionally compact; export is not. The HTML and JSON downloads contain every retained frame, VM event, memory sample, argument, startup milestone, and explicit drop count. The exported trace is a faithful VM-event projection for Chrome/DevTools timeline import. It does not invent CPU stacks, GPU counters, thread identity, or source locations that the captured data does not contain. maxEvents, frame retention, and memory sampling are the only bounded limits, and their drops remain visible in the export.

Run #

Run with Flutter's normal integration-test commands:

flutter test integration_test/task_flow_test.dart -d <device>

For Cockpit-managed development sessions, the same test can run from the direct application project or optional development module and its steps remain visible in the session timeline and artifacts. Case/Suite documents remain available for AI-generated, black-box, matrix, and cross-platform journeys; this package is the ergonomic Dart layer for Flutter source projects.