SDPM Chunking

Sentence-level chunking with semantic double-pass merging

SDPM (Semantic Double-Pass Merging) combines sentence-level chunking with semantic analysis for optimal chunk boundaries.

Quick Start

Agent with SDPM Chunking

from praisonaiagents import Agent

agent = Agent(
    instructions="Answer questions from research papers.",
    knowledge={
        "sources": ["papers/"],
        "chunker": {
            "type": "sdpm",
            "chunk_size": 512,
            "embedding_model": "all-MiniLM-L6-v2"
        }
    }
)

response = agent.start("Summarize the findings")

from praisonaiagents.knowledge.chunking import Chunking

chunker = Chunking(
    chunker_type="sdpm",
    chunk_size=512,
    embedding_model="all-MiniLM-L6-v2"
)

chunks = chunker.chunk("Your research paper content...")

When to Use

• Research papers with complex structure
• Technical documents with multiple topics
• Content where both sentence flow AND semantic coherence matter

Parameters

Parameter	Type	Default	Description
chunk_size	int	512	Max tokens per chunk
embedding_model	str	auto	Model for semantic analysis