Chunking Strategies Overview

Understanding document chunking strategies for optimal RAG performance

PraisonAI integrates chonkie, a high-performance chunking library, to provide flexible document processing strategies.

Quick Start

Agent with Chunking

from praisonaiagents import Agent

# Agent with semantic chunking
agent = Agent(
    instructions="Answer questions from documents.",
    knowledge={
        "sources": ["research.pdf"],
        "chunker": {
            "type": "semantic",
            "chunk_size": 512
        }
    }
)

response = agent.start("What are the key findings?")

from praisonaiagents.knowledge.chunking import Chunking

# Create a chunker
chunker = Chunking(
    chunker_type="recursive",
    chunk_size=512,
    chunk_overlap=128
)

# Chunk some text
chunks = chunker.chunk("Your long document text here...")
for chunk in chunks:
    print(chunk.text)

Available Strategies

Strategy	Alias	Best For	Speed
Token	token	Fixed-size chunks	⚡ Fast
Sentence	sentence	Natural boundaries	⚡ Fast
Recursive	recursive	Structured documents	⚡ Fast
Semantic	semantic	Topic segmentation	🔄 Medium
SDPM	sdpm	Research papers	🔄 Medium
Late	late	Better embeddings	🔄 Medium

Choosing a Strategy

Agent Configuration

Simplest (Default Strategy)

from praisonaiagents import Agent

# Uses token chunking by default
agent = Agent(
    instructions="Answer from documents.",
    knowledge=["docs/"]  # Default chunking
)

With Chunking Config

from praisonaiagents import Agent

agent = Agent(
    instructions="Answer from documents.",
    knowledge={
        "sources": ["research.pdf", "data/"],
        "chunker": {
            "type": "semantic",       # Strategy type
            "chunk_size": 512,        # Tokens per chunk
            "chunk_overlap": 128,     # Overlap between chunks
            "embedding_model": "all-MiniLM-L6-v2"  # For semantic/sdpm/late
        }
    }
)

All Chunker Options

Option	Type	Default	Description
type	str	"token"	Chunker type: token, sentence, recursive, semantic, sdpm, late
chunk_size	int	512	Target tokens per chunk
chunk_overlap	int	128	Overlap between chunks
tokenizer_or_token_counter	str	"gpt2"	Tokenizer for counting
embedding_model	str	auto	Embedding model (semantic/sdpm/late only)

Strategy Details

Token Chunking

Fixed-size token chunks. Fast and predictable.

Sentence Chunking

Split at sentence boundaries. Natural flow.

Recursive Chunking

Hierarchical splitting. Great for markdown.

Semantic Chunking

Similarity-based splits. Topic coherence.

Installation

Chunking requires the knowledge extra:

pip install "praisonaiagents[knowledge]"

This installs the chonkie library automatically.