tokens property

List<String> get tokens

Extracted semantic tokens from raw input.

Implementation

List<String> get tokens =>
    rawInput.toLowerCase().split(RegExp(r'[\s,;.!?]+'))
      ..removeWhere((t) => t.isEmpty);