flutter_hand_gesture

Real-time hand gesture recognition for Flutter — fully on-device, no internet required, no API key needed.

Built with ❤️ by KalaiNova Infotech

---

Features

• ✋ 10 built-in gestures — fist, open palm, thumbs up/down, peace, pointing, rock, call me, OK, vulcan
• 🎯 21-point hand landmarks — full skeleton tracking
• 🧠 Custom gesture training — teach your own gestures with KNN classifier
• 📡 Stream-based API — reactive, works with BLoC / Provider / Riverpod
• 🔇 Fully offline — TFLite models run on-device
• 🎨 Plug-and-play widget — GestureCameraView with landmark overlay
• 🤲 Multi-hand support — detect up to 2 hands simultaneously
• ⚡ Background isolate — zero UI thread blocking
• 🪔 Tamil / ISL sign presets — 15 Tamil signs + 5 vowel fingerspelling, matched by shape, orientation & motion
• 🔗 Gesture sequences — recognize ordered combos (e.g. 👍→✌️→🖐 = unlock) with progress tracking
• 🎬 Gesture → action mapper — accessibility & presentation presets, plus your own bindings

---

Quick Start

// 1. Plug-and-play widget
GestureCameraView(
  onGestureDetected: (gesture) {
    print('${gesture.emoji} ${gesture.displayName}');
    // 👍 Thumbs Up
  },
)

// 2. Stream-based (headless)
final detector = HandGestureDetector(
  config: GestureConfig(minConfidence: 0.8),
);
await detector.initialize();
await detector.startDetection();

detector.gestureStream.listen((result) {
  final gesture = result.primaryGesture;
  if (gesture != null) {
    print(gesture.type); // GestureType.thumbsUp
  }
});

// 3. Custom gesture training
final trainer = CustomGestureTrainer();
await trainer.load();

// Training (collect 5+ samples per gesture)
await trainer.addSample('namaste', landmarks);

// Prediction
final label = trainer.predict(landmarks); // 'namaste'

// 4. Tamil / ISL sign recognition (shape + orientation + motion)
final recognizer = SignRecognizer<TamilSign>(TamilSignPreset.signs);

// Feed every frame (use debounceDuration: Duration.zero so motion flows through)
final sign = recognizer.update(landmarks, DateTime.now());
if (sign != null) {
  print('${sign.emoji} ${sign.tamilScript} (${sign.meaning})');
  // 👎 கெட்ட (Bad) — thumbs-down, told apart from thumbs-up நல்ல
}

// 5. Gesture sequences — ordered combos as one action
final sequences = GestureSequenceRecognizer()
  ..registerAll(GestureSequenceRecognizer.defaults);

sequences.bind(detector.gestureStream).listen((matched) {
  print('Sequence: ${matched.label}'); // e.g. 'Unlock'
});

// 6. Map gestures to app actions
final mapper = GestureActionMapper.presentation();
final action = mapper.resolve(GestureType.pointing); // 'next_slide'

---

Supported Gestures

Gesture	Emoji	GestureType
Fist	✊	GestureType.fist
Open Palm	🖐	GestureType.openPalm
Thumbs Up	👍	GestureType.thumbsUp
Thumbs Down	👎	GestureType.thumbsDown
Peace	✌️	GestureType.peace
Pointing	👆	GestureType.pointing
Rock Sign	🤘	GestureType.rockSign
Call Me	🤙	GestureType.callMe
OK Sign	👌	GestureType.okSign
Vulcan	🖖	GestureType.vulcanSalute
Custom	🤚	GestureType.custom

---

Installation

dependencies:
  flutter_hand_gesture: ^2.1.0

Android

Add to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />

Set minSdkVersion to 24 in android/app/build.gradle(.kts) (required by the on-device ML backend):

minSdk = 24

iOS

Add to ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>Camera is used for hand gesture recognition</string>

---

Configuration

GestureConfig(
  minConfidence: 0.75,        // 0.0 - 1.0
  maxHands: 1,                // 1 or 2
  showLandmarks: true,        // draw skeleton overlay
  targetGestures: [           // filter to specific gestures only
    GestureType.thumbsUp,
    GestureType.peace,
  ],
  debounceDuration: Duration(milliseconds: 500),
  mirrorCamera: true,         // for front camera
  useBackgroundIsolate: true, // keep UI smooth
)

---

API Reference

HandGestureDetector

Method	Description
initialize()	Set up camera and ML model
startDetection()	Begin processing frames
stopDetection()	Pause processing
gestureStream	Stream<GestureResult>
landmarkStream	Stream<List<HandLandmarks>>
dispose()	Release all resources

GestureCameraView

Prop	Type	Description
config	GestureConfig	Detection settings
onGestureDetected	(HandGesture) → void	Called on each gesture
onResult	(GestureResult) → void	Full frame result
showGestureLabel	bool	Show label overlay
showLandmarks	bool	Show skeleton overlay
overlayBuilder	Widget Function(context, result)	Custom overlay

CustomGestureTrainer

Method	Description
addSample(label, landmarks)	Add training sample
predict(landmarks)	Predict label
predictWithConfidence(landmarks)	Predict with score
save()	Persist to disk
load()	Load from disk
clearLabel(label)	Remove a gesture

SignRecognizer<T extends SignPattern> (v2.1.0)

Matches Tamil/ISL signs from live landmarks using finger shape + hand orientation + motion, so signs sharing a shape (thumbs-up vs thumbs-down, static vs wagging index) no longer collide.

Member	Description
SignRecognizer(signs)	Build over TamilSignPreset.signs or IslAlphabetPreset.signs
update(landmarks, timestamp)	Feed a frame; returns the matched sign or null
isWagging	Whether the index finger is currently wagging
reset()	Clear the motion buffer (e.g. hand left frame)

Feed every frame — use GestureConfig(debounceDuration: Duration.zero) so motion is detectable.

Sign presets (v2.0.0)

Preset	Contents
TamilSignPreset	15 Tamil signs (vanakkam, nandri, kaadhal…) with Tamil script, meaning, emoji
IslAlphabetPreset	ISL fingerspelling for 5 Tamil vowels (அ ஆ இ ஈ உ)

Each sign carries a fingerPattern plus optional orientation (HandOrientation) and motion (SignMotion). loadInto(trainer) and getSign(name) are also available.

GestureSequenceRecognizer (v2.0.0)

Recognizes ordered gesture combos as a single named action.

Member	Description
register(seq) / registerAll(seqs)	Add sequences
update(gesture, timestamp)	Feed a gesture; returns a completed GestureSequence or null
bind(gestureStream)	Pipe a stream → broadcast Stream<GestureSequence>
progress / totalSteps	Live step counts for UI feedback
defaults	Predefined unlock / screenshot / dismiss

Pair with SequenceProgressPainter to draw ● ● ○ ○ step dots.

GestureActionMapper (v2.0.0)

Member	Description
on(gesture, action) / onSequence(name, action)	Bind to an action string
resolve(gesture) / resolveSequence(name)	Look up the action, or null
accessibility() / presentation()	Ready-made mappings

---

Roadmap

• x TFLite MediaPipe model integration (v0.2.0) — via hand_detection
• x Tamil / ISL sign language preset (v2.0.0)
• x Gesture sequence recognition (v2.0.0)
• x Orientation + motion disambiguation (v2.1.0) — SignRecognizer
• Web platform support
• macOS / Windows support

---

License

MIT © KalaiNova Infotech