encode abstract method

TokenizedInput encode(
  1. String prefix,
  2. String text
)

Tokenizes (prefix + text) per this tokenizer's convention (BOS/EOS wrap for SentencePiece, CLS/.../SEP for WordPiece).

Implementation

TokenizedInput encode(String prefix, String text);