> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Token Utilities Module

> Token counting and auto-chunking utilities for LLM evaluation

# Token Utilities

Lightweight token counting and context-aware chunking utilities. Works without external dependencies, with optional accuracy boost from `litellm`.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
flowchart LR
    subgraph Input
        A[Text Content]
    end
    
    subgraph TokenUtils["Token Utilities"]
        B[estimate_tokens]
        C[count_tokens]
        D[get_context_length]
        E[needs_chunking]
    end
    
    subgraph Output
        F{Chunk?}
        G[Process Whole]
        H[Split & Process]
    end
    
    A --> C
    C --> E
    D --> E
    E --> F
    F -->|No| G
    F -->|Yes| H
    
    style A fill:#8B0000,color:#fff
    style G fill:#8B0000,color:#fff
    style H fill:#8B0000,color:#fff
    style B fill:#189AB4,color:#fff
    style C fill:#189AB4,color:#fff
    style D fill:#189AB4,color:#fff
    style E fill:#189AB4,color:#fff
```

## Quick Start

```python Basic Usage theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import estimate_tokens, needs_chunking

# Estimate tokens (no dependencies)
tokens = estimate_tokens("Hello world")
print(tokens)  # 2

# Check if chunking needed
if needs_chunking(large_text, model="gpt-4o-mini"):
    # Split into chunks
    pass
```

```python With Details theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import needs_chunking

# Get detailed chunking info
info = needs_chunking(
    text=my_content,
    model="gpt-4o-mini",
    safety_margin=0.8,
    return_info=True
)

print(f"Needs chunking: {info['needs_chunking']}")
print(f"Token count: {info['estimated_tokens']}")
print(f"Context window: {info['context_length']}")
print(f"Utilization: {info['utilization']:.1%}")
```

## Functions

### estimate\_tokens

Estimate token count without external dependencies.

> [!TIP]
>
> Uses OpenAI's heuristics: \~4 characters per token, \~0.75 tokens per word.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import estimate_tokens

# Default (conservative - uses max of char/word methods)
tokens = estimate_tokens("Hello world")  # 2

# Character-based only
tokens = estimate_tokens("Hello world", method="chars")  # 2

# Word-based only  
tokens = estimate_tokens("Hello world", method="words")  # 2

# Average of both methods
tokens = estimate_tokens("Hello world", method="average")  # 2
```

**text** `str` *required*

Text to estimate tokens for

**method** `str` (default: max)

Estimation method: `chars`, `words`, `max`, `min`, `average`

### count\_tokens

Count tokens accurately using `litellm` if available, otherwise estimate.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import count_tokens

# Uses litellm for accuracy (if installed)
tokens = count_tokens("Hello world", model="gpt-4o-mini")  # 2

# Force estimation (skip litellm)
tokens = count_tokens("Hello world", use_litellm=False)  # 2
```

**text** `str` *required*

Text to count tokens for

**model** `str` (default: gpt-4o-mini)

Model name for tokenizer selection

**use\_litellm** `bool` (default: True)

Whether to try litellm for accurate counting

> [!NOTE]
>
> For very large text (>100K chars), automatically falls back to estimation for performance.

### get\_context\_length

Get the context window size for a model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import get_context_length

# OpenAI — new 1M-context families
print(get_context_length("gpt-5"))            # 1047576
print(get_context_length("gpt-4.1"))          # 1047576

# OpenAI — existing 128K family
print(get_context_length("gpt-4o-mini"))      # 128000
print(get_context_length("gpt-4o"))           # 128000

# Anthropic (dated snapshot still resolvable via eval extras)
print(get_context_length("claude-3-5-sonnet-20241022"))  # 200000

# Google
print(get_context_length("gemini-1.5-pro"))   # 2097152

# Versioned name → partial match resolves to the specific entry, not the shorter prefix
print(get_context_length("gpt-4-32k-0314"))   # 32768  (not 8192)
```

> [!NOTE]
>
> Lookups use partial matching by descending key length, so a specific versioned name (e.g. `gpt-4-32k-0314`) resolves to the more specific entry (`gpt-4-32k` → 32768) before falling back to a shorter prefix (`gpt-4` → 8192). The shared budgeter path consults litellm's `model_cost` registry first — the 128,000-token default only applies when both litellm and this static table miss. See [How the context window is resolved](https://praison.ai/docs/features/context-budgeter.md#how-the-context-window-is-resolved).

**model** `str` *required*

Model name (e.g., "gpt-4o-mini", "claude-3-5-sonnet-20241022")

**use\_litellm** `bool` (default: True)

Whether to try litellm for accurate info

> [accordion] Supported Models
>
> The eval path resolves the union of the canonical `MODEL_LIMITS` and the eval-only `_EVAL_EXTRA_LENGTHS`.
>
> | Model | Context Window |
> | - | - |
> | gpt-5, gpt-5-mini, gpt-5-nano | 1,047,576 |
> | gpt-4.1, gpt-4.1-mini, gpt-4.1-nano | 1,047,576 |
> | gpt-4o, gpt-4o-mini | 128,000 |
> | gpt-4-turbo, gpt-4-turbo-preview | 128,000 |
> | gpt-4 | 8,192 |
> | gpt-4-32k | 32,768 |
> | gpt-3.5-turbo, gpt-3.5-turbo-16k | 16,385 |
> | o3, o3-mini, o4-mini | 200,000 |
> | o1 | 200,000 |
> | o1-mini, o1-preview | 128,000 |
> | claude-3-5-sonnet, claude-3-5-haiku | 200,000 |
> | claude-3-opus, claude-3-sonnet, claude-3-haiku | 200,000 |
> | claude-3-5-sonnet-20241022, claude-3-5-sonnet-latest | 200,000 |
> | claude-3-5-haiku-20241022 | 200,000 |
> | claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307 | 200,000 |
> | claude-2.1 | 200,000 |
> | claude-2 | 100,000 |
> | gemini-1.5-pro | 2,097,152 |
> | gemini-1.5-flash, gemini-1.5-flash-8b | 1,048,576 |
> | gemini-2.0-flash, gemini-2.0-flash-exp | 1,048,576 |
> | gemini-pro | 32,760 |
> | mistral-large-latest | 128,000 |
> | mistral-medium-latest, mistral-small-latest, codestral-latest | 32,000 |
> | deepseek-chat, deepseek-coder | 64,000 |
> | llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.1-8b-instant | 128,000 |
> | mixtral-8x7b-32768 | 32,768 |

### needs\_chunking

Determine if text needs to be chunked for the given model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import needs_chunking

# Simple check
if needs_chunking(large_text, "gpt-4o-mini"):
    print("Text too large, needs chunking")

# With custom safety margin (70% of context)
needs_chunk = needs_chunking(
    text=large_text,
    model="gpt-4o-mini",
    safety_margin=0.7
)

# Get detailed info
info = needs_chunking(large_text, "gpt-4o-mini", return_info=True)
# Returns: {
#   'needs_chunking': True,
#   'estimated_tokens': 150000,
#   'context_length': 128000,
#   'available_tokens': 102400,
#   'safety_margin': 0.8,
#   'utilization': 1.17
# }
```

**text** `str` *required*

Text to evaluate

**model** `str` (default: gpt-4o-mini)

Model name to check context window for

**safety\_margin** `float` (default: 0.8)

Fraction of context window to use (leaves room for prompts)

**return\_info** `bool` (default: False)

If True, return detailed info dict instead of bool

### get\_recommended\_chunk\_size

Get recommended chunk size in characters for a model.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.eval import get_recommended_chunk_size

# Default (5 chunks)
chunk_size = get_recommended_chunk_size("gpt-4o-mini")  # ~10000

# More chunks for finer granularity
chunk_size = get_recommended_chunk_size("gpt-4o-mini", target_chunks=10)
```

**model** `str` (default: gpt-4o-mini)

Model name

**target\_chunks** `int` (default: 5)

Target number of chunks

**safety\_margin** `float` (default: 0.8)

Fraction of context to use

## CLI Integration

Use auto-chunking with the recipe judge command:

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Auto-detect if chunking is needed
praisonai recipe judge run-abc123 --auto-chunk

# Force chunked evaluation
praisonai recipe judge run-abc123 --chunked --chunk-size 8000
```

**1. Run Recipe**

Execute your recipe to generate a trace

**2. Judge with Auto-Chunk**

Use `--auto-chunk` to automatically handle large outputs

**3. Review Results**

Get accurate evaluation even for large content

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
flowchart TD
    A[Input Text] --> B{litellm available?}
    B -->|Yes| C[litellm.token_counter]
    B -->|No| D[estimate_tokens]
    C --> E[Token Count]
    D --> E
    
    F[Model Name] --> G{litellm available?}
    G -->|Yes| H[litellm.model_cost]
    G -->|No| I[DEFAULT_CONTEXT_LENGTHS]
    H --> J[Context Length]
    I --> J
    
    E --> K{tokens > context × margin?}
    J --> K
    K -->|Yes| L[Chunk Content]
    K -->|No| M[Process Whole]
    
    style A fill:#8B0000,color:#fff
    style L fill:#8B0000,color:#fff
    style M fill:#8B0000,color:#fff
    style C fill:#189AB4,color:#fff
    style D fill:#189AB4,color:#fff
    style H fill:#189AB4,color:#fff
    style I fill:#189AB4,color:#fff
```

> [accordion] Token Estimation Heuristics
>
> Based on OpenAI's guidance for English text:
>
> * **1 token ≈ 4 characters**
> * **1 token ≈ 0.75 words**
> * **100 tokens ≈ 75 words**
>
> The `max` method (default) uses the larger of character-based and word-based estimates for conservative results.

## Related

* [Eval Module](https://praison.ai/docs/sdk/praisonaiagents/eval/eval.md) - Evaluation framework
* [Recipe Judge CLI](https://praison.ai/docs/cli/eval.md) - CLI evaluation commands


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.