Open
Description
I am wondering if there is an appetite for adding the support for counting tokens.
With increase of generative AI for coding, being able to count the number of tokens seems a much needed feature.
There's a crate could be used for this: tiktoken-rs
Metadata
Metadata
Assignees
Labels
No labels