fuzzy_duplicate_detector 🔍

An ultra-pro Dart/Flutter package for detecting fuzzy duplicate records — even when names are spelled differently, carry diacritics, mix Arabic and Latin scripts, or have words in different order.

The Problem

Every real-world Arabic-language database eventually looks like this:

Record in DB	Same person?
`محمد علي`	✅ Yes
`محمد عَلي`	✅ Yes (diacritics)
`M.ALI`	✅ Yes (transliterated + punctuation)
`علي محمد`	✅ Yes (word-order swap)
`أحمد`	❌ No

fuzzy_duplicate_detector solves this with a multi-layer algorithm stack that handles all of these cases simultaneously.

Features

Feature	Description
🔤 Arabic Soundex	Phonetic encoding designed for Arabic consonants
🔤 English Soundex	Russell & Odell Soundex for Latin names
✏️ Levenshtein Distance	Unicode-safe edit-distance similarity
🏁 Jaro-Winkler	Prefix-weighted similarity (great for names)
🔀 Token-Set Ratio	Word-order-independent matching
🌍 Transliteration	Arabic ↔ Latin cross-script phonetic matching
🧹 Arabic Normalizer	Diacritics, Alef variants, Teh Marbuta, Tatweel
🧹 Latin Normalizer	Accents, punctuation, case folding
🔗 Union-Find Clustering	Transitive grouping (A≈B, B≈C → {A,B,C} one group)
⚡ Soundex-Blocked Mode	O(n·log n) for datasets > 500 items
⚙️ Fully Configurable	Threshold, weights, preprocessing toggles
🎛️ 3 Built-in Presets	`strict`, `lenient`, `crossScript`

Installation

dependencies:
  fuzzy_duplicate_detector: ^1.0.0

dart pub get

Quick Start

import 'package:fuzzy_duplicate_detector/fuzzy_duplicate_detector.dart';

void main() {
  final groups = FuzzyDedup.find([
    'محمد علي',
    'محمد عَلي',    // diacritics
    'M.ALI',         // transliterated
    'علي محمد',      // word order swapped
    'أحمد',          // different person
  ]);

  for (final g in groups) {
    print('Canonical: ${g.canonical}');
    print('Members  : ${g.members}');
    print('Confidence: ${g.confidencePercent}');
  }
  // Canonical: محمد علي
  // Members  : [محمد علي, محمد عَلي, M.ALI, علي محمد]
  // Confidence: 91.4%
}

API Reference

`FuzzyDedup` — Static Methods

`FuzzyDedup.find(items, {config})`

Find all duplicate groups in a list of strings.

final groups = FuzzyDedup.find(names);
// Returns: List<DuplicateGroup>
// Groups sorted by averageConfidence descending.
// Items with no duplicates are NOT included.

`FuzzyDedup.compare(a, b, {config})`

Compare a single pair and return full details.

final r = FuzzyDedup.compare('مصطفى', 'مصطفي');
print(r.confidencePercent);    // e.g. "97.3%"
print(r.levenshteinScore);     // 0.875
print(r.soundexMatch);         // true
print(r.toDetailedString());   // full multi-line breakdown

`FuzzyDedup.deduplicate(items, {config})`

Return a list with duplicates replaced by their canonical form.

final clean = FuzzyDedup.deduplicate([
  'محمد علي', 'محمد عَلي', 'خالد',
]);
// → ['محمد علي', 'خالد']

`FuzzyDedup.findAsMap(items, {config})`

Return Map<canonical, List<non-canonical duplicates>>.

final map = FuzzyDedup.findAsMap(names);
// { 'محمد علي': ['محمد عَلي', 'M.ALI'] }

`FuzzyDedup.findUniqueItems(items, {config})`

Return only items that are NOT in any duplicate group.

`FuzzyDedup.groupSummary(items, {config})`

Return a human-readable text report of all groups.

print(FuzzyDedup.groupSummary(names));
// Found 2 duplicate group(s) in 6 item(s):
//
//   Group 1  [avg: 93.1%]
//     Canonical : "محمد علي"
//     Duplicate : "محمد عَلي"  (96.8%)
//     Duplicate : "M.ALI"  (89.4%)

Configuration

FuzzyDedup.find(
  names,
  config: DedupConfig(
    threshold:               0.85,   // 85% minimum confidence
    removeArabicDiacritics:  true,   // محمد عَلي == محمد علي
    normalizeArabicAlef:     true,   // أحمد == احمد
    normalizeArabicTehMarbuta: true, // فاطمة == فاطمه
    normalizeArabicAlefMaqsura: true,// مصطفى == مصطفي
    enableTransliteration:   true,   // محمد ≈ Mohamed
    clusteringThreshold:     500,    // switch to blocked mode at 500 items
    weights: AlgorithmWeights(
      levenshtein: 0.35,
      jaroWinkler:  0.30,
      soundex:      0.20,
      tokenSet:     0.15,
    ),
  ),
);

Built-in Presets

Preset	Threshold	Use Case
`DedupConfig()`	0.80	General purpose (default)
`DedupConfig.strict`	0.92	Banking KYC, regulatory compliance
`DedupConfig.lenient`	0.65	Exploratory analysis, human review
`DedupConfig.crossScript`	0.72	Arabic ↔ Latin matching

Algorithm Weight Presets

Preset	Best For
`AlgorithmWeights()`	General Arabic/English names
`AlgorithmWeights.phoneticHeavy`	Noisy, OCR-scanned data
`AlgorithmWeights.editDistanceHeavy`	Clean, typed data

Result Objects

`MatchResult`

final r = FuzzyDedup.compare('Abdullah', 'عبدالله');

r.a                    // 'Abdullah'
r.b                    // 'عبدالله'
r.confidence           // 0.813
r.confidencePercent    // '81.3%'
r.levenshteinScore     // 0.0 (different scripts)
r.jaroWinklerScore     // 0.0 (different scripts)
r.soundexA             // 'A430'
r.soundexB             // 'ع590'
r.soundexMatch         // false
r.soundexScore         // 0.0
r.tokenSetScore        // 0.0
r.transliterationBonus // 0.122  ← cross-script bonus!
r.isDuplicate          // true   (with threshold 0.60)

`DuplicateGroup`

final g = groups.first;

g.canonical           // 'محمد علي'   (longest member)
g.members             // ['محمد علي', 'محمد عَلي', 'M.ALI']
g.size                // 3
g.averageConfidence   // 0.914
g.confidencePercent   // '91.4%'
g.maxConfidence       // 0.968
g.minConfidence       // 0.861
g.nonCanonicalMembers // ['محمد عَلي', 'M.ALI']
g.pairwiseResults     // List<MatchResult>

Algorithm Stack — How It Works

Input: ["محمد علي", "محمد عَلي", "M.ALI", "أحمد"]
              │
              ▼
    ┌─────────────────────────┐
    │  GroupBuilder           │  ← orchestrates the pipeline
    └────────┬────────────────┘
             │  For each pair (i, j):
             ▼
    ┌──────────────────────────────────────────────┐
    │           ScoreCombiner.compare(a, b)         │
    │                                              │
    │  1. Normalize → ArabicNormalizer             │
    │                 LatinNormalizer              │
    │                                              │
    │  2. Levenshtein.similarity()  → levScore     │
    │                                              │
    │  3. JaroWinkler.similarity()  → jwScore      │
    │                                              │
    │  4. ArabicSoundex.encode()    → codeA, codeB │
    │     EnglishSoundex.encode()   (auto-detect)  │
    │     soundexScore = 1.0 / 0.75 / 0.5 / 0.0   │
    │                                              │
    │  5. Transliterator.arabicToLatin()           │
    │     (cross-script bonus, max +0.15)          │
    │                                              │
    │  6. Token-Set Ratio          → tokenScore    │
    │     (sort tokens, then JW)                   │
    │                                              │
    │  7. confidence = Σ(wᵢ × scoreᵢ) + bonus     │
    └──────────────────┬───────────────────────────┘
                       │  if confidence ≥ threshold
                       ▼
              UnionFind.union(i, j)
                       │
                       ▼
              UnionFind.getGroups()
                       │
                       ▼
    ┌──────────────────────────────────────────┐
    │  Output: List<DuplicateGroup>            │
    │  sorted by averageConfidence desc        │
    └──────────────────────────────────────────┘

Performance

Dataset Size	Mode	Typical Time (M1 Mac)
50 names	Full O(n²)	~2 ms
200 names	Full O(n²)	~30 ms
500 names	Full O(n²)	~180 ms
1000 names	Soundex-blocked	~40 ms
5000 names	Soundex-blocked	~180 ms

Run the bundled benchmark:

dart run benchmark/large_dataset_benchmark.dart

Running Tests

dart test

The test suite covers:

All 7 FuzzyDedup static methods
All Arabic normalisation steps
Latin normalisation (accents, punctuation, ß→ss)
Transliterator script detection
Levenshtein distance (classical + Arabic)
English Soundex (Robert/Rupert, Mohamed/Muhammad)
Arabic Soundex (مصطفى/مصطفي, diacritics invariance)
Jaro-Winkler (MARTHA/MARHTA, prefix bonus)
AlgorithmWeights sum validation
UnionFind (transitive grouping, path compression)
DedupConfig (presets, copyWith)

Use Cases

🏦 Banking KYC — detect duplicate customer onboarding applications
🏢 CRM systems — merge duplicate contact records
🏛️ Government databases — national ID / voter registry deduplication
🛒 E-commerce — merge duplicate customer accounts
📊 Data warehouses — ETL pipeline deduplication
🏥 Healthcare — patient record deduplication (HIPAA contexts)
📚 Libraries — author name deduplication in bibliographic data

Dependencies

Package	Version	Purpose
`characters`	^1.3.0	Unicode grapheme-cluster iteration
`collection`	^1.18.0	`groupBy`, `ListEquality`

Both are official Dart team packages — minimal footprint, no transitive deps.

Contributing

Pull requests are welcome! Please:

Fork the repository
Create a feature branch: git checkout -b feature/my-feature
Add tests for new functionality
Run dart test and ensure all tests pass
Run dart analyze and fix any warnings
Submit a PR against main

fuzzy_duplicate_detector 🔍

The Problem

Features

Installation

Quick Start

API Reference

`FuzzyDedup` — Static Methods

`FuzzyDedup.find(items, {config})`

`FuzzyDedup.compare(a, b, {config})`

`FuzzyDedup.deduplicate(items, {config})`

`FuzzyDedup.findAsMap(items, {config})`

`FuzzyDedup.findUniqueItems(items, {config})`

`FuzzyDedup.groupSummary(items, {config})`

Configuration

Built-in Presets

Algorithm Weight Presets

Result Objects

`MatchResult`

`DuplicateGroup`

Algorithm Stack — How It Works

Performance

Running Tests

Use Cases

Dependencies

Contributing

License

Libraries

fuzzy_duplicate_detector package

fuzzy_duplicate_detector 🔍

The Problem

Features

Installation

Quick Start

API Reference

FuzzyDedup — Static Methods

FuzzyDedup.find(items, {config})

FuzzyDedup.compare(a, b, {config})

FuzzyDedup.deduplicate(items, {config})

FuzzyDedup.findAsMap(items, {config})

FuzzyDedup.findUniqueItems(items, {config})

FuzzyDedup.groupSummary(items, {config})

Configuration

Built-in Presets

Algorithm Weight Presets

Result Objects

MatchResult

DuplicateGroup

Algorithm Stack — How It Works

Performance

Running Tests

Use Cases

Dependencies

Contributing

License

Libraries

fuzzy_duplicate_detector package

`FuzzyDedup` — Static Methods

`FuzzyDedup.find(items, {config})`

`FuzzyDedup.compare(a, b, {config})`

`FuzzyDedup.deduplicate(items, {config})`

`FuzzyDedup.findAsMap(items, {config})`

`FuzzyDedup.findUniqueItems(items, {config})`

`FuzzyDedup.groupSummary(items, {config})`

`MatchResult`

`DuplicateGroup`