> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunking Strategies

> Document chunking strategies for optimal RAG performance using the chonkie library

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="researcher",
    instructions="Use chunked knowledge for long documents",
    knowledge=True,
)

agent.start("Find the section about deployment limits")
```

The user queries long sources; chunking controls what reaches retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Chunking"
        Document[📋 Document] --> Chunker[✂️ Chunking Engine]
        Chunker --> VectorDB[💾 Vector DB]
        VectorDB --> Retrieval[✅ Retrieval]
    end

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef tool fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef store fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef success fill:#10B981,stroke:#7C90A0,color:#fff
    class Document input
    class Chunker tool
    class VectorDB store
    class Retrieval success
```

PraisonAI integrates [chonkie](https://github.com/chonkie-inc/chonkie) for high-performance document chunking.

## Quick Start

**1. Default chunking on an agent**

```python Agent with Chunking (Simplest) theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

# Default chunking (token-based)
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge=["research.pdf", "docs/"]
)

response = agent.start("What are the key findings?")
```

```python Agent with Custom Chunking theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    instructions="Answer questions from documents.",
    knowledge={
        "sources": ["research.pdf"],
        "chunker": {
            "type": "semantic",       # token, sentence, recursive, semantic, sdpm, late
            "chunk_size": 512,
            "chunk_overlap": 128
        }
    }
)

response = agent.start("What methodology was used?")
```

```python Direct Chunking API theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Chunking

# Create chunker
chunker = Chunking(
    chunker_type="recursive",
    chunk_size=512,
    chunk_overlap=128
)

# Chunk text
chunks = chunker.chunk("Your document content here...")
for chunk in chunks:
    print(f"Tokens: {chunk.token_count}")
    print(chunk.text[:100])
```

## Available Strategies

| Strategy | Best For | Speed |
| - | - | - |
| `token` | Fixed-size chunks | ⚡ Fastest |
| `sentence` | Natural boundaries | ⚡ Fast |
| `recursive` | Structured docs (markdown) | ⚡ Fast |
| `semantic` | Topic segmentation | 🔄 Medium |
| `sdpm` | Research papers | 🔄 Medium |
| `late` | Best embeddings | 🔄 Medium |

> [card] [Token Chunking](https://praison.ai/docs/features/rag/strategies/token)
>
> Fixed-size token chunks. Fast and predictable.

> [card] [Sentence Chunking](https://praison.ai/docs/features/rag/strategies/sentence)
>
> Split at sentence boundaries. Natural flow.

> [card] [Recursive Chunking](https://praison.ai/docs/features/rag/strategies/recursive)
>
> Hierarchical splitting. Great for markdown.

> [card] [Semantic Chunking](https://praison.ai/docs/features/rag/strategies/semantic)
>
> Similarity-based splits. Topic coherence.

## Chunker Configuration

> [!NOTE]
>
> Chunker settings (`chunk_size`, `chunk_overlap`, `type`) apply uniformly to every supported file type — PDFs, DOCX, media via MarkItDown, source files, and plain-text files (`.md`, `.txt`, `.csv`, `.json`, `.xml`, `.html`, `.htm`). On pre-[#4831](https://github.com/MervinPraison/PraisonAI/pull/4831) releases these settings were silently ignored for the plain-text extensions.

### All Parameters

| Parameter | Type | Default | Applies To |
| - | - | - | - |
| `type` | str | `"token"` | All |
| `chunk_size` | int | 512 | All |
| `chunk_overlap` | int | 128 | token, sentence |
| `tokenizer_or_token_counter` | str | `"gpt2"` | token, sentence, recursive |
| `embedding_model` | str | auto | semantic, sdpm, late |

### Strategy Examples

> [tab] Token
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> agent = Agent(
>     instructions="Process documents.",
>     knowledge={
>         "sources": ["docs/"],
>         "chunker": {
>             "type": "token",
>             "chunk_size": 256,
>             "chunk_overlap": 50
>         }
>     }
> )
> ```

> [tab] Sentence
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> agent = Agent(
>     instructions="Process articles.",
>     knowledge={
>         "sources": ["articles/"],
>         "chunker": {
>             "type": "sentence",
>             "chunk_size": 512
>         }
>     }
> )
> ```

> [tab] Recursive
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> agent = Agent(
>     instructions="Process markdown docs.",
>     knowledge={
>         "sources": ["README.md", "docs/"],
>         "chunker": {
>             "type": "recursive",
>             "chunk_size": 512
>         }
>     }
> )
> ```

> [tab] Semantic
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> agent = Agent(
>     instructions="Process research papers.",
>     knowledge={
>         "sources": ["papers/"],
>         "chunker": {
>             "type": "semantic",
>             "chunk_size": 512,
>             "embedding_model": "all-MiniLM-L6-v2"
>         }
>     }
> )
> ```
>
> > [!NOTE]
> >
> > `semantic`, `sdpm`, and `late` chunkers lazily import `chonkie.embeddings.AutoEmbeddings` on the first lookup. Since PraisonAI PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) a missing `chonkie` install raises `ImportError: chonkie package not found. Please install it using: pip install 'praisonaiagents[knowledge]'` — a broken transitive dependency (e.g. `sentence-transformers`) keeps its original error unchanged.

## Which Strategy Should I Use?

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Start([What's your priority?]) --> B{Speed or quality?}
    B -->|Speed| C[token<br/>fast, fixed-size]
    B -->|Quality| D{Document type?}
    D -->|Structured/Markdown| E[recursive]
    D -->|Natural prose| F{Topic boundaries?}
    F -->|Yes| G[semantic]
    F -->|No| H[sentence]

    classDef start fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef answer fill:#10B981,stroke:#7C90A0,color:#fff

    class Start start
    class B,D,F question
    class C,E,G,H answer
```

***

## How It Works

The user adds documents; the chunking engine splits them by the chosen strategy, embeds each chunk, and stores it for retrieval.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Chunker
    participant VectorDB

    User->>Agent: knowledge=["docs/"]
    Agent->>Chunker: Split documents
    Chunker->>VectorDB: Store embedded chunks
    User->>Agent: start("Find deployment limits")
    Agent->>VectorDB: Retrieve relevant chunks
    VectorDB-->>User: Answer from chunks
```

## Installation

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
pip install "praisonaiagents[knowledge]"
```

This installs the chonkie library automatically.

## Best Practices

> [accordion] Match chunk size to retrieval use case
>
> Smaller chunks improve precision for Q\&A; larger chunks preserve narrative for summarisation tasks.

> [accordion] Install the knowledge extra
>
> Run `pip install "praisonaiagents[knowledge]"` so chonkie and indexing backends load only when needed.

> [accordion] Test overlap on sample docs
>
> Tune overlap on a representative document before indexing an entire corpus.

> [accordion] Pair with smart retrieval
>
> Combine chunking with hybrid search and reranking for better recall on long knowledge bases.

## Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| `ImportError: chonkie package not found. Please install it using: pip install 'praisonaiagents[knowledge]'` on the first `Knowledge.add(...)` / `agent.start(...)` call | `semantic` (the default), `sdpm`, or `late` chunker is selected but the `[knowledge]` extra is not installed. Since PraisonAI PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) both the chunker registry and the embedding backend surface the same actionable hint. | `pip install 'praisonaiagents[knowledge]'`. Or fall back to `chunking_strategy="sentence"` / `"paragraph"` / `"fixed"` / `"token"`, which do not need chonkie's embedding model. |
| `ImportError` that names `chonkie.embeddings` but **omits** the `pip install` hint | `chonkie` is installed but its embeddings submodule cannot import (broken `sentence-transformers`, `torch`, or `transformers`). PR [#5105](https://github.com/MervinPraison/PraisonAI/pull/5105) intentionally preserves the original error so real transitive failures are not masked. | Read the underlying error, then reinstall or upgrade that package (`pip install --upgrade sentence-transformers` is the most common fix). |

## Related

> [card] [Knowledge Base](https://praison.ai/docs/features/knowledge.md)
>
> Configure knowledge sources and retrieval

> [card] [RAG Agents](https://praison.ai/docs/features/rag.md)
>
> Build retrieval-augmented agents

> [card] [Bot Platform Capabilities](https://praison.ai/docs/features/bot-platform-capabilities.md)
>
> How platform capabilities drive this feature


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.