Token Estimation

Fast offline token counting for context management

from praisonaiagents import Agent

agent = Agent(name="token-agent", instructions="Estimate token counts accurately.")
agent.start("Estimate how many tokens this message will consume.")

Token Estimation

PraisonAI provides fast, offline token estimation that works without API calls. This enables real-time context budget tracking and optimization decisions.

The user estimates tokens for a prompt; heuristics predict cost before the agent calls the model.

How It Works

Quick Start

1. Estimate text tokens

   from praisonaiagents import estimate_tokens_heuristic
   
   tokens = estimate_tokens_heuristic("Hello, how are you today?")
   print(f"Estimated: {tokens} tokens")
   
2. Estimate messages and tool schemas

   from praisonaiagents import estimate_messages_tokens, estimate_tool_schema_tokens
   
   messages = [
       {"role": "system", "content": "You are helpful."},
       {"role": "user", "content": "What is Python?"},
   ]
   print(f"Messages: {estimate_messages_tokens(messages)} tokens")
   

Estimation Algorithm

The heuristic estimator uses character-based rules optimized for typical LLM tokenization:

Character Type	Tokens per Character
ASCII text	~0.25 (4 chars = 1 token)
Non-ASCII (Unicode)	~1.3 tokens per char
Whitespace	Counted normally

Message Overhead

Each message includes overhead for role markers and formatting:

• Base overhead: 4 tokens per message
• Role tokens: ~2 tokens
• Content: Estimated via heuristic

API Reference

estimate_tokens_heuristic(text: str) -> int

Estimate tokens for a string using character-based heuristics.

tokens = estimate_tokens_heuristic("Hello world!")
# Returns: ~3 tokens

estimate_messages_tokens(messages: List[Dict]) -> int

Estimate total tokens for a list of chat messages.

messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hi there!"},
]
tokens = estimate_messages_tokens(messages)

estimate_tool_schema_tokens(tools: List[Dict]) -> int

Estimate tokens for tool/function schemas.

tools = [{"name": "search", "description": "Search the web"}]
tokens = estimate_tool_schema_tokens(tools)

TokenEstimatorImpl

Class-based estimator with caching:

from praisonaiagents import TokenEstimatorImpl

estimator = TokenEstimatorImpl()
tokens = estimator.estimate("Some text")

get_estimator() -> TokenEstimatorImpl

Get a singleton estimator instance:

from praisonaiagents import get_estimator

estimator = get_estimator()
tokens = estimator.estimate("Text to estimate")

Accuracy Considerations

The heuristic estimator is designed for speed over perfect accuracy:

Scenario	Accuracy
English text	~90-95%
Code	~85-90%
Mixed content	~85-90%
Non-ASCII heavy	~80-85%

For budget decisions, the estimator adds a small safety margin to prevent underestimation.

Performance

• Speed: < 1ms for 100K characters
• Memory: O(1) - no caching required
• No API calls: Works completely offline

Integration with Budgeter

from praisonaiagents import (
    ContextBudgeter,
    estimate_messages_tokens,
)

budgeter = ContextBudgeter(model="gpt-4o-mini")
budget = budgeter.allocate()

# Check if messages fit in budget
messages = [...]  # Your conversation
tokens = estimate_messages_tokens(messages)

if tokens > budget.usable * 0.8:
    print("Warning: Approaching context limit!")

Best Practices

Estimate before each LLM call

Check message totals against the budget to trigger compaction proactively.

Use model-aware estimators

Pass the model name so tiktoken or provider-specific counting applies when available.

Warn at 80% utilisation

Act before hard limits — retrieval trimming and summarisation need headroom.

Validate estimates in CI

Run validated mode on fixture conversations to catch drift in token counting.

Related

Context Ledger

Track tokens by segment

Context Budgeter

Allocate token budgets