Chunking

Text chunking utilities for RAG and document processing

Split long documents into chunks for RAG pipelines and knowledge bases.

Quick Start

1. Simple Usage

   npm install praisonai
   

   import { Chunking } from 'praisonai';
   
   const chunker = new Chunking({ chunkSize: 500, overlap: 50 });
   const chunks = chunker.chunk('Your long text here...');
   
2. With Configuration

   const chunker = new Chunking({ chunkSize: 500, overlap: 50, strategy: 'sentence' });
   const chunks = chunker.chunkBySentence('First sentence. Second sentence.');
   

---

Basic Usage

import { Chunking } from 'praisonai';

const chunker = new Chunking({ chunkSize: 500, overlap: 50 });
const chunks = chunker.chunk('Your long text here...');

chunks.forEach(chunk => {
  console.log(`Chunk ${chunk.index}: ${chunk.content.substring(0, 50)}...`);
});

Chunking Strategies

By Size

import { Chunking } from 'praisonai';

const chunker = new Chunking({
  chunkSize: 500,
  overlap: 50,
  strategy: 'size'
});

const text = 'A'.repeat(1500);
const chunks = chunker.chunkBySize(text);
console.log(`Created ${chunks.length} chunks`);

By Sentence

import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'sentence' });

const text = 'First sentence. Second sentence. Third sentence.';
const chunks = chunker.chunkBySentence(text);

chunks.forEach(chunk => {
  console.log(chunk.content);
});

By Paragraph

import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'paragraph' });

const text = `Paragraph one.

Paragraph two.

Paragraph three.`;

const chunks = chunker.chunkByParagraph(text);
console.log(`Found ${chunks.length} paragraphs`);

Semantic Chunking

import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'semantic' });

const text = `# Header 1
Content under header 1.

# Header 2
Content under header 2.`;

const chunks = chunker.chunkBySemantic(text);

Chunk Structure

interface Chunk {
  content: string;      // Chunk text content
  index: number;        // Chunk index
  startOffset: number;  // Start position in original
  endOffset: number;    // End position in original
  metadata?: Record<string, any>;
}

Merge Small Chunks

import { Chunking } from 'praisonai';

const chunker = new Chunking({ strategy: 'sentence' });
const chunks = chunker.chunkBySentence(text);

// Merge chunks smaller than 100 characters
const merged = chunker.mergeSmallChunks(chunks, 100);

Factory Function

import { createChunking } from 'praisonai';

const chunker = createChunking({
  chunkSize: 1000,
  overlap: 100,
  strategy: 'size'
});

Integration with Knowledge Base

import { Knowledge, Chunking } from 'praisonai';

const kb = new Knowledge();
const chunker = new Chunking({ strategy: 'size', chunkSize: 512, overlap: 50 });

const document = 'Your long document text...';

for (const chunk of chunker.chunk(document)) {
  await kb.store(chunk.content, {
    userId: 'u1',
    metadata: { startOffset: chunk.startOffset },
  });
}

kb.add('doc.pdf') already chunks internally using the chunker block of KnowledgeStoreConfig. Hand-chunk and store() only when you want to control the chunks yourself.

Related

Chunking CLI

CLI knowledge commands

Knowledge Base

Store chunked documents

RAG Agent

Retrieval workflows