classify_card #

On-device document handling for Flutter, in three independent pieces:

Use any one of them alone. They share no state, no platform interface and no thread, so the gate works without the cropper and the cropper works without the model.

The gate and the cropper run entirely on the device. Nothing is uploaded, and no network permission is needed.

What it does and does not do #

It is a pre-check and a cropper, not a replacement for your classify/OCR backend.

Requirements #

Still on API 21–22 or iOS 12? Pin classify_card: ^1.0.0. It has the classifier and the realtime scanner view, and neither needed the higher floors. Version 2.0.0 is a major bump for these two lines alone — every existing call still compiles unchanged.

On iOS, Info.plist needs NSCameraUsageDescription for CardGateScannerView.

You do not need to add noCompress 'tflite', register Xcode resources, or ship the model yourself — the plugin carries its own model and loads it in a way that does not depend on host-app packaging settings.

Install #

dependencies:
  classify_card: ^2.0.0

One host-app change is required.

Android — android/app/build.gradle:

android {
  defaultConfig {
    minSdkVersion 21   // was flutter.minSdkVersion / 19
  }
}

iOS — nothing. The pod declares static_framework = true, so it installs under a plain use_frameworks! alongside other TFLite-based pods.

Only if you use the scanner (CardGateScannerView), add the camera purpose string to ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>The camera is used to scan your identity document.</string>

Only if your app gets camera from a git fork — ekyc_flutter_sdk does — pub cannot reconcile a git source with a hosted one. Point both at the same fork:

dependency_overrides:
  camera:
    git:
      url: https://github.com/Techainer/flutter-camera-package.git
      ref: lucas_handle_resolution_prest
      path: packages/camera/camera

Then rebuild the app — this plugin contains native code, so a hot restart is not enough.

Quick start #

import 'package:classify_card/classify_card.dart';

final gate = ClassifyCard();

// Once, e.g. in initState of your capture screen. Never throws.
await gate.warmUp();

// After the user captures or picks a photo.
final decision = await gate.evaluate(photoPath, mode: CardGateMode.enforce);

if (!decision.allow) {
  showRetakeGuidance();   // no backend call
  return;
}

await myBackend.classify(photoPath);   // your existing flow, unchanged

photoPath should be the whole photo, upright. Do not crop it to the card first — the model was trained on full frames. A file:// prefix is fine, as is a bare path; Android also accepts a content:// URI. EXIF rotation is handled for you.

Realtime scanner #

CardGateScannerView is a camera screen with a card-shaped guide that scores frames on-device as they arrive. The guide recolours as the model changes its mind; the user presses the shutter; you get the photo back through a callback and do whatever you want with it.

Navigator.of(context).push(MaterialPageRoute(
  builder: (context) => CardGateScannerView(
    onCaptured: (capture) async {
      Navigator.of(context).pop();

      // capture.path is a JPEG on disk. What happens next is entirely yours.
      final decision = await ClassifyCard()
          .evaluate(capture.path, mode: CardGateMode.enforce);
      if (decision.allow) await myBackend.classify(capture.path);
    },
    onCancel: () => Navigator.of(context).pop(),
  ),
));

Useful knobs: autoCapture + autoCaptureFrames to take the photo after N consecutive card frames, minFrameInterval (default 250 ms) to trade responsiveness for battery, overlay to restyle the guide, and the four label strings.

Three things worth knowing #

A frame is a hint, not a verdict. capture.liveResult is the last frame the model scored. Do not act on it. The photo the shutter produced is what you run evaluate on - that goes through the still-photo path the model was calibrated against.

The model sees the whole frame, not the guide. The guide helps the user aim; cropping to it would break the full-frame assumption the model was trained on, so the plugin does not.

Dart never touches a pixel. The plane bytes go straight from the camera to native code, which does the colour conversion, the rotation and the resize. Nothing is encoded to JPEG or PNG and nothing is written to disk, which is what makes several checks a second affordable.

Frames that arrive while one is still being scored are dropped, not queued - queueing at frame rate grows without bound and ends in an out-of-memory kill. The scanner ignores those silently.

If you want the live scoring without the UI, call classifyFrame yourself:

controller.startImageStream((image) async {
  if (busy) return;
  busy = true;
  try {
    final result = await gate.classifyFrame(
      CardGateFrame.fromCameraImage(image, rotationDegrees: description.sensorOrientation),
    );
    setState(() => status = result.label);
  } on CardGateException catch (e) {
    if (e.code != CardGateException.codeBusy) status = null;  // a dropped frame is normal
  } finally {
    busy = false;
  }
});

Cropping #

// No UI: a file goes in, a cropped file comes out.
final result = await DocumentCropper().crop(photoPath);
await myBackend.upload(result.path);   // always a usable path

It runs DocAligner's lcnet100_h_e_bifpn_256 corner model (Apache-2.0, bundled at assets/document_corners_lcnet100.onnx, 4.5 MB) through ONNX Runtime, and warps the photo to the four corners it returns. It has nothing to do with ML Kit.

final result = await DocumentCropper().crop(path);

switch (result.reason) {
  case DocumentCropReason.cropped:           // result.path is a new file
  case DocumentCropReason.alreadyFullFrame:  // the photo was already just the document
  case DocumentCropReason.noDocument:        // nothing found; result.path is your original
  case DocumentCropReason.unavailable:
  case DocumentCropReason.timeout:
  case DocumentCropReason.failed:
}

result.path is always safe to use. When nothing was cropped it is the path you passed in, byte-for-byte unchanged, and result.cropped is false. crop never throws. That asymmetry is deliberate: a wrong crop destroys information permanently, a missing crop only fails to add any.

result.detector says which engine produced the corners — model, builtin (the geometric fallback, used only when the model cannot load), vision, or systemScanner.

How good it is. Measured against ground-truth corners on 24 real ID-card photographs plus 6 taken on a phone: median IoU 0.958, one photograph off by more than 30 px, and nothing cropped on the six photographs that contain no document at all. About 30 ms on a mid-range phone.

For comparison, the geometric detector this package shipped before shows median IoU 0.872, four photographs off by more than 30 px, and found nothing at all in any of the six phone photographs the model handles. It is still in the package, but only as the fallback for when the model cannot load.

What it costs. ONNX Runtime adds roughly 14 MB of native library per ABI, and the model 4.5 MB. ONNX Runtime ships arm only — there is no x86/x86_64 build — so this path does not run on an x86 Android emulator. Use an arm emulator or a real device.

Why this package carries its own corner model #

Because it cannot. The entire public surface of the ML Kit document scanner is:

GmsDocumentScanning.getClient(options)           // -> GmsDocumentScanner
GmsDocumentScanner.getStartScanIntent(Activity)  // -> Task<IntentSender>
GmsDocumentScanningResult.fromActivityResultIntent(Intent)

One method, which takes an Activity and returns an IntentSender. There is no overload accepting a Bitmap, Uri, File or InputImage, so it can only ever open its own camera. Apple's VNDocumentCameraViewController is a view controller, with the same consequence.

"Photo in, cropped photo out" is a different problem, and neither platform ships an answer to it that takes a file. Verified three ways against the 16.0.0 AAR: no method accepts an image, there is no input extra, and the only IPC surface is com.google.mlkit.vision.docscan.**ui**.aidls.* — the model and the UI both live inside Play Services, reachable only by launching that UI.

So this package carries its own model instead: DocAligner's corner network, Apache-2.0, bundled here and run through ONNX Runtime. Nothing in this package depends on ML Kit or VisionKit.

The two rules that matter #

1. It fails open #

decision.allow is false for exactly one reason: CardGateReason.other, meaning the model is confident there is no document. Every other outcome lets the request through — a missing plugin, a corrupt model, a decode error, a timeout. A broken classifier must never block a customer holding a valid card.

reason.isFailure is true for the last three. Log those — a spike means something is broken.

2. Start in shadow, not enforce #

mode defaults to CardGateMode.shadow: the model runs and reports, but never blocks. Ship that first, log the results next to what your backend decides, and only switch to CardGateMode.enforce once the two agree on your own traffic.

final decision = await gate.evaluate(path);         // shadow — cannot reject anyone
final r = decision.result;
if (r != null) {
  analytics.log('card_gate', r.toMap());            // safe to log: scores only, never the image
}

API #

Three classes, three independent features. None of them touches the others.

ClassifyCard — the gate #

Future<String?> warmUp();
Future<CardGateResult> classify(String imagePath, {Duration timeout});
Future<CardGateDecision> evaluate(String imagePath, {CardGateMode mode, Duration timeout});
Future<CardGateResult> classifyFrame(CardGateFrame frame, {Duration timeout});
Future<void> release();

CardGateResult carries label, score (calibrated), rawScore (the model's own output), modelVersion, latencyMs and decodeMs. toMap() is safe to log — it never contains image data.

Do not re-threshold rawScore yourself. 0.5 is not the production operating point. score is the calibrated value that label was derived from, using the Platt transform and threshold in the bundled policy.

DocumentCropper — photo in, cropped photo out #

Future<DocumentCropResult> crop(
  String imagePath, {
  int quality = 95,
  Duration timeout = const Duration(seconds: 8),
  bool preferPlatformDetector = false,
});
Future<void> clearCache();

DocumentCropResult carries path, cropped, reason, detector, width, height, latencyMs, engineNote (why the requested engine did not run, when it did not), and corners — the four detected points as fractions of the source photo, ordered top-left, top-right, bottom-right, bottom-left. toMap() is safe to log.

preferPlatformDetector lets Apple's VNDetectDocumentSegmentationRequest try first on iOS 15+. It is off by default so both platforms run the same model and agree on the same photo.

Logging #

The plugin deliberately does no logging of its own. It hands you the data; you decide what goes to your own telemetry. Nothing it returns contains image bytes.

How each platform runs the model #

Both platforms run the same pipeline: EXIF-upright, bilinear resize to 224x224 with half-pixel centres, the metadata normalization, then the input tensor's own quantization. The arithmetic is identical on both, so a photo should score the same either side.

Neither uses the TFLite Task Library, which is what the vendor reference is written against. org.tensorflow:tensorflow-lite-task-vision was never ported to LiteRT — there is no litert-task-vision — and shipping it alongside LiteRT puts two TFLite runtimes in one APK. On iOS TensorFlowLiteTaskVision is unusable in a host app for three separate reasons:

• its 0.4.3 static framework exports the same TfLite* symbols as TensorFlowLiteC, so it collides with any app that already ships TFLite;
• it redefines MLKit's GMLImage class, so it collides with any app using MLKit;
• it has no arm64 simulator slice, so it cannot build for the iOS Simulator on Apple Silicon;
• it has not been released since November 2022.

The consequence you should know about: the preprocessing is a faithful reimplementation, not the vendor's own code, so rawScore can differ slightly from the reference implementation. Measure it on your own images before enforcing — this is the main reason to run shadow first.

Verifying it before you enforce #

1. Run the example/ app, set the mode selector to shadow.
2. Feed it ~10 real documents (including rotated 90° / 180° / 270°) and ~10 non-documents — portraits, landscapes, junk.
3. Check the label is right in every case, and compare rawScore between an Android device and an iPhone on the same photos.
4. Watch latencyMs. Target is p95 under 100 ms after warm-up.
5. Only then switch to enforce.

Integrity #

The model is verified on every startup: its SHA-256 must equal metadata.model_sha256 in the bundled policy.json, and the policy's schema_version, score_transform.kind, scale and threshold are all range-checked. A mismatch makes the classifier unavailable — which fails open, it does not silently reject photos. The bundled model hashes to 361bbfc4e540e717d6a961ed119f47719416241ed939b2e0e90024ca14dd5855.

Model tensors, for reference: input input_1 UINT8 [1,224,224,3] scale 0.003921568859368563 zero-point 0; output Identity UINT8 [1,2] scale 0.00390625 zero-point 0.

Troubleshooting #

reason is always unavailable

Either the plugin is not in the binary — do a full rebuild, not a hot restart — or the native setup failed. Check the device log for card_gate_unavailable; the message names the exact cause (missing resource, SHA-256 mismatch, bad policy field).

Android build fails on minSdkVersion

Raise minSdkVersion to at least 21 in your app's android/app/build.gradle.

CocoaPods cannot resolve TensorFlowLiteSwift

This pod asks for ~> 2.12.0 (>= 2.12.0, < 2.13.0), matching ekyc_flutter_sdk and tflite_flutter. If another pod in your app needs a different 2.x, CocoaPods cannot install both — one version of a pod is all it will take. Align the constraints, then re-run the verification steps above.

Note that changing a development pod's constraint is not picked up by pod install; run pod update TensorFlowLiteSwift to make CocoaPods re-resolve it.

transitive dependencies that include statically linked binaries

You should not see this: the podspec declares static_framework = true, so a plain use_frameworks! is enough. If it appears anyway, another pod depending on TensorFlowLiteSwift is missing that declaration.

Duplicate symbol errors mentioning TfLite

Something else in your app links its own TFLite copy — most often TensorFlowLiteTaskVision, MediaPipeTasksVision, or a vendored .framework. Only one TFLite runtime can be linked; consolidate on TensorFlowLiteSwift.

Android: duplicate libtensorflowlite_jni.so

Another dependency pulls org.tensorflow:tensorflow-lite* while this plugin uses LiteRT. They are different Maven coordinates, so Gradle keeps both. Move the other dependency to com.google.ai.edge.litert:*.

iOS Simulator on an Apple Silicon Mac

Works. TensorFlowLiteSwift ships a proper xcframework with an arm64 simulator slice — this is one of the reasons Task Vision was rejected.

License #

See LICENSE. The bundled model and policy are covered by your agreement with the model provider, not by this package's license.