genUiFuzz function

Future<List<GenUiFuzzFinding>> genUiFuzz({
  1. required Catalog catalog,
  2. required GenUiPump pump,
  3. Set<String> only = const <String>{},
  4. Set<String> skip = const <String>{},
  5. int maxCasesPerComponent = 80,
  6. bool reportEmpty = false,
})

Renders everything catalog's own examples allow, mutated, and reports what broke.

A catalog is a contract with something that cannot be recompiled, and the only thing holding a model to it is a JSON schema. Every widget test writes the input the author had in mind; this writes the inputs the schema permits and the author did not. A required property the model left out, a string where a number was declared, a binding that never resolves, a list with two hundred entries: each of those is a message a real agent can send, and each one renders here before it renders in front of someone.

The cases are built by mutating each item's own generated example rather than by reading the schema, because the example is valid by construction and already names every required property. That also means a hand-written CatalogItem is fuzzed as well as a generated one, as long as it carries example data.

testWidgets('the catalog survives what its schema allows', (tester) async {
  final findings = await genUiFuzz(
    catalog: exampleCatalog,
    pump: tester.pumpWidget,
  );
  expect(findings, isEmpty, reason: findings.join('\n\n'));
});

reportEmpty also reports a case that rendered nothing at all. It is off by default because most of those are correct: take the items away from a list and it has nothing to draw. Turn it on when you want to find the component that swallows a wrong-typed property and leaves a blank space where the agent asked for something.

skip leaves components out by name, for the ones whose failure you have already accepted. only restricts the run to a few while you are fixing one. maxCasesPerComponent bounds a component with many properties; the cases are generated in a fixed order, so a run is reproducible and a bound always drops the same tail.

Implementation

Future<List<GenUiFuzzFinding>> genUiFuzz({
  required Catalog catalog,
  required GenUiPump pump,
  Set<String> only = const <String>{},
  Set<String> skip = const <String>{},
  int maxCasesPerComponent = 80,
  bool reportEmpty = false,
}) async {
  final String? catalogId = catalog.catalogId;
  if (catalogId == null) {
    throw ArgumentError.value(
      catalog,
      'catalog',
      'has no catalogId, and a surface names the catalog it was built '
          'against. Give the catalog an id before fuzzing it.',
    );
  }

  final findings = <GenUiFuzzFinding>[];
  var surface = 0;

  for (final CatalogItem item in catalog.items) {
    if (skip.contains(item.name)) continue;
    if (only.isNotEmpty && !only.contains(item.name)) continue;

    final List<JsonMap>? example = _parseExample(item);
    if (example == null) continue;

    final int rootIndex = example.indexWhere((c) => c['id'] == 'root');
    final JsonMap root = example[rootIndex < 0 ? 0 : rootIndex];
    final int index = rootIndex < 0 ? 0 : rootIndex;

    // The untouched example first: it is the baseline every case is compared
    // against, and a component whose own example throws needs no fuzzing to
    // be worth reporting.
    final _Render baseline = await _render(
      pump,
      catalog,
      catalogId,
      example,
      's${surface++}',
    );
    if (baseline.error != null) {
      findings.add(
        GenUiFuzzFinding(
          component: item.name,
          mutation: 'its own generated example',
          kind: GenUiFuzzKind.threw,
          detail: baseline.error!,
          componentJson: jsonEncode(root),
        ),
      );
      continue;
    }

    for (final _Case c in _casesFor(root).take(maxCasesPerComponent)) {
      final List<JsonMap> components = <JsonMap>[...example];
      components[index] = c.component;

      final _Render r = await _render(
        pump,
        catalog,
        catalogId,
        components,
        's${surface++}',
      );
      if (r.error != null) {
        findings.add(
          GenUiFuzzFinding(
            component: item.name,
            mutation: c.label,
            kind: GenUiFuzzKind.threw,
            detail: r.error!,
            componentJson: jsonEncode(c.component),
          ),
        );
      } else if (reportEmpty && baseline.hasSize && !r.hasSize) {
        findings.add(
          GenUiFuzzFinding(
            component: item.name,
            mutation: c.label,
            kind: GenUiFuzzKind.renderedNothing,
            detail: 'the untouched example took up space and this took up none',
            componentJson: jsonEncode(c.component),
          ),
        );
      }
    }
  }

  return findings;
}