tokenizeFts function

List<String> tokenizeFts(
  1. String text
)

Split text into lower-cased word tokens. The same tokenizer is used for both indexing and query terms.

Implementation

List<String> tokenizeFts(String text) {
  final out = <String>[];
  final re = RegExp(r'[A-Za-z0-9_]+');
  for (final m in re.allMatches(text)) {
    out.add(m.group(0)!.toLowerCase());
  }
  return out;
}