What
The truncating text feature could make use of a --verbose option to display if text has been truncated (has exceeded the truncation limit), and maybe how many tokens are left.
Why
It can be useful to know if the text has effectively been truncated, as a warning that not all text has passed through, and maybe another pass is needed for the rest of the text.
I can see people using ttok as safeguard before all their LLM API calls to avoid hitting the API tokens limits. When that limit is high and not expected to be hit, having a warning when it does will come in handy.
Notes
The logged warning should not be part of the stdout of ttok which is expected to be piped. Maybe stderr would fit more here.
What
The truncating text feature could make use of a
--verboseoption to display if text has been truncated (has exceeded the truncation limit), and maybe how many tokens are left.Why
It can be useful to know if the text has effectively been truncated, as a warning that not all text has passed through, and maybe another pass is needed for the rest of the text.
I can see people using
ttokas safeguard before all their LLM API calls to avoid hitting the API tokens limits. When that limit is high and not expected to be hit, having a warning when it does will come in handy.Notes
The logged warning should not be part of the stdout of
ttokwhich is expected to be piped. Maybe stderr would fit more here.