Documentation
¶
Overview ¶
Package token estimates LLM token counts from byte content using a characters-per-token heuristic.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EstimateTokens ¶
EstimateTokens provides a rough token count estimate for content.
Uses a simple heuristic of ~4 characters per token for English text. This is a conservative estimate for Claude/GPT-style tokenizers that tends to slightly overestimate, which is safer for budgeting.
Parameters:
- content: Byte slice to estimate tokens for
Returns:
- int: Estimated token count (0 for empty content)
func EstimateTokensString ¶
EstimateTokensString estimates tokens for a string.
Convenience wrapper around EstimateTokens for string input.
Parameters:
- s: String to estimate tokens for
Returns:
- int: Estimated token count
Types ¶
This section is empty.
Source Files
Click to show internal directories.
Click to hide internal directories.