flutter_hand_gesture #

Real-time hand gesture recognition for Flutter — fully on-device, no internet required, no API key needed.

Built with ❤️ by KalaiNova Infotech

Features #

• ✋ 10 built-in gestures — fist, open palm, thumbs up/down, peace, pointing, rock, call me, OK, vulcan
• 🎯 21-point hand landmarks — full skeleton tracking
• 🧠 Custom gesture training — teach your own gestures with KNN classifier
• 📡 Stream-based API — reactive, works with BLoC / Provider / Riverpod
• 🔇 Fully offline — TFLite models run on-device
• 🎨 Plug-and-play widget — GestureCameraView with landmark overlay
• 🤲 Multi-hand support — detect up to 2 hands simultaneously
• ⚡ Background isolate — zero UI thread blocking
• 🪔 Tamil / ISL sign presets — 15 Tamil signs + 5 vowel fingerspelling, matched by shape, orientation & motion
• 🔗 Gesture sequences — recognize ordered combos (e.g. 👍→✌️→🖐 = unlock) with progress tracking
• 🎬 Gesture → action mapper — accessibility & presentation presets, plus your own bindings

Quick Start #

// 1. Plug-and-play widget
GestureCameraView(
  onGestureDetected: (gesture) {
    print('${gesture.emoji} ${gesture.displayName}');
    // 👍 Thumbs Up
  },
)

// 2. Stream-based (headless)
final detector = HandGestureDetector(
  config: GestureConfig(minConfidence: 0.8),
);
await detector.initialize();
await detector.startDetection();

detector.gestureStream.listen((result) {
  final gesture = result.primaryGesture;
  if (gesture != null) {
    print(gesture.type); // GestureType.thumbsUp
  }
});

// 3. Custom gesture training
final trainer = CustomGestureTrainer();
await trainer.load();

// Training (collect 5+ samples per gesture)
await trainer.addSample('namaste', landmarks);

// Prediction
final label = trainer.predict(landmarks); // 'namaste'

// 4. Tamil / ISL sign recognition (shape + orientation + motion)
final recognizer = SignRecognizer<TamilSign>(TamilSignPreset.signs);

// Feed every frame (use debounceDuration: Duration.zero so motion flows through)
final sign = recognizer.update(landmarks, DateTime.now());
if (sign != null) {
  print('${sign.emoji} ${sign.tamilScript} (${sign.meaning})');
  // 👎 கெட்ட (Bad) — thumbs-down, told apart from thumbs-up நல்ல
}

// 5. Gesture sequences — ordered combos as one action
final sequences = GestureSequenceRecognizer()
  ..registerAll(GestureSequenceRecognizer.defaults);

sequences.bind(detector.gestureStream).listen((matched) {
  print('Sequence: ${matched.label}'); // e.g. 'Unlock'
});

// 6. Map gestures to app actions
final mapper = GestureActionMapper.presentation();
final action = mapper.resolve(GestureType.pointing); // 'next_slide'

Supported Gestures #

Installation #

dependencies:
  flutter_hand_gesture: ^2.1.0

Android #

Add to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />

Set minSdkVersion to 24 in android/app/build.gradle(.kts) (required by the on-device ML backend):

minSdk = 24

iOS #

Add to ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>Camera is used for hand gesture recognition</string>

Configuration #

GestureConfig(
  minConfidence: 0.75,        // 0.0 - 1.0
  maxHands: 1,                // 1 or 2
  showLandmarks: true,        // draw skeleton overlay
  targetGestures: [           // filter to specific gestures only
    GestureType.thumbsUp,
    GestureType.peace,
  ],
  debounceDuration: Duration(milliseconds: 500),
  mirrorCamera: true,         // for front camera
  useBackgroundIsolate: true, // keep UI smooth
)

API Reference #

HandGestureDetector #

GestureCameraView #

CustomGestureTrainer #

SignRecognizer<T extends SignPattern> (v2.1.0) #

Matches Tamil/ISL signs from live landmarks using finger shape + hand orientation + motion, so signs sharing a shape (thumbs-up vs thumbs-down, static vs wagging index) no longer collide.

Feed every frame — use GestureConfig(debounceDuration: Duration.zero) so motion is detectable.

Sign presets (v2.0.0) #

Each sign carries a fingerPattern plus optional orientation (HandOrientation) and motion (SignMotion). loadInto(trainer) and getSign(name) are also available.

GestureSequenceRecognizer (v2.0.0) #

Recognizes ordered gesture combos as a single named action.

Pair with SequenceProgressPainter to draw ● ● ○ ○ step dots.

GestureActionMapper (v2.0.0) #

Roadmap #

• ✅ TFLite MediaPipe model integration (v0.2.0) — via hand_detection
• ✅ Tamil / ISL sign language preset (v2.0.0)
• ✅ Gesture sequence recognition (v2.0.0)
• ✅ Orientation + motion disambiguation (v2.1.0) — SignRecognizer
• ❌ Web platform support
• ❌ macOS / Windows support

License #

MIT © KalaiNova Infotech