targetTokens property

int? targetTokens
final

The target number of tokens to keep. The default value is trigger_tokens/2.

Discarding parts of the context window causes a temporary latency increase so this value should be calibrated to avoid frequent compression operations.

Implementation

final int? targetTokens;