> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental Indexing

> Index only changed files — skip unchanged content to keep knowledge bases fast and up to date

Skip unchanged files automatically — only modified content is re-indexed on each run, saving time on large document corpora.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="DocExpert",
    instructions="Answer questions using the knowledge base.",
    knowledge=["./docs"],
)

agent.start("What changed in the docs since last week?")
```

The user queries the knowledge base; only changed files are re-indexed on each run.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    A[📁 Files] --> B[🔍 FileTracker]
    B --> C{Changed?}
    C -->|Yes| D[📥 Index]
    C -->|No| E[⏭ Skip]
    D --> F[📊 IndexResult]
    E --> F

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef process fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef result fill:#10B981,stroke:#7C90A0,color:#fff

    class A input
    class B,C,D,E process
    class F result

```

## Quick Start

**1. Agent with incremental knowledge**

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent

agent = Agent(
    name="DocExpert",
    instructions="Answer questions using the knowledge base.",
    knowledge=["./docs"],
    memory={"user_id": "my_user"},
)

response = agent.start("What are the main topics covered?")
```

On the second run, unchanged files are skipped automatically.

**2. Direct index with result stats**

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent, Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./docs",
    memory={"user_id": "my_user"},
    incremental=True,
)

print(f"Indexed: {result.files_indexed}, Skipped: {result.files_skipped}")
print(f"Duration: {result.duration_seconds:.2f}s")
```

***

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant FileTracker
    participant Knowledge

    User->>Agent: start("What changed?")
    Agent->>Knowledge: index("./docs", incremental=True)
    Knowledge->>FileTracker: load state from .praison/.index_state.json
    loop For each file
        Knowledge->>FileTracker: has_changed(filepath)?
        alt File changed or new
            FileTracker-->>Knowledge: True
            Knowledge->>Knowledge: delete stale chunks (from previous memory_ids)
            Knowledge->>Knowledge: index file
            Knowledge->>FileTracker: mark_indexed(filepath, memory_ids=[...])
        else File unchanged
            FileTracker-->>Knowledge: False
            Knowledge->>Knowledge: skip
        end
    end
    Knowledge->>FileTracker: save state
    Knowledge-->>Agent: IndexResult
    Agent-->>User: answer
```

| Component | Purpose |
| - | - |
| `FileTracker` | Tracks file hash and mtime across sessions |
| `IndexResult` | Reports indexed, skipped, and error counts |
| `CorpusStats` | Corpus-level statistics and strategy recommendation |
| `.praisonignore` | Gitignore-style patterns to exclude files |

***

## Configuration Options

### `Knowledge.index()` parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `path` | `str` | — | Directory or file path to index |
| `incremental` | `bool` | `True` | Skip unchanged files |
| `force` | `bool` | `False` | Re-index all files regardless of changes |
| `include_glob` | `list[str]` | `None` | Glob patterns to include, e.g. `["*.md", "*.py"]` |
| `exclude_glob` | `list[str]` | `None` | Glob patterns to exclude, e.g. `["*.log", "test_*"]` |
| `user_id` | `str` | `None` | Scope index to a user |
| `agent_id` | `str` | `None` | Scope index to an agent |
| `run_id` | `str` | `None` | Scope index to a run |

### `IndexResult` fields

| Field | Type | Default | Description |
| - | - | - | - |
| `success` | `bool` | `True` | `False` when any file failed to index (i.e. `errors` is non-empty). A partial failure — even one file out of many — counts as `success=False`, since a knowledge base quietly missing a document is the loss this flag exists to surface. |
| `files_indexed` | `int` | `0` | Files newly indexed |
| `files_skipped` | `int` | `0` | Files skipped (unchanged) |
| `chunks_created` | `int` | `0` | Chunks created from indexed files |
| `errors` | `list[str]` | `[]` | Error messages for failed files |
| `duration_seconds` | `float` | `0.0` | Total indexing time |
| `corpus_stats` | `CorpusStats \| None` | `None` | Corpus statistics |

> [!NOTE]
>
> **Partial failures count.** If nine files out of ten embed successfully and one fails, `result.success` is `False` and `result.errors` names the one that failed. `files_indexed` still reports the nine that landed, so the caller can log both the loss and the win.

### `CorpusStats` fields

| Field | Type | Default | Description |
| - | - | - | - |
| `file_count` | `int` | `0` | Number of files in corpus |
| `chunk_count` | `int` | `0` | Number of chunks created |
| `total_tokens` | `int` | `0` | Estimated total tokens |
| `indexed_at` | `str \| None` | `None` | ISO timestamp of last index |
| `path` | `str \| None` | `None` | Path to corpus root |
| `strategy_recommendation` | `str` | computed | Retrieval strategy based on corpus size |

### `FileTracker` API

`FileTracker` records which vector-store chunks belong to each file so a re-index can remove stale content first.

| Method | Signature | Description |
| - | - | - |
| `mark_indexed` | `mark_indexed(filepath, info, memory_ids=None)` | Record a file as indexed. `memory_ids` are the vector-store IDs produced for that file, so the next re-index knows which chunks to delete. |
| `get_memory_ids` | `get_memory_ids(filepath) -> list[str]` | Return the memory IDs previously stored for a file (empty list if none). |

**Why this matters:** editing a file and re-indexing no longer leaves the old chunks behind, so search results reflect only the current content. IDs that fail to delete are retried on the next re-index — they are never orphaned.

> [!NOTE]
>
> **What gets deleted** — inspect the chunks that will be removed when a file changes:
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> from praisonaiagents import FileTracker
>
> tracker = FileTracker(state_file=".praison/.index_state.json")
> tracker.load()
>
> print(tracker.get_memory_ids("./docs/readme.md"))
> # ['mem_abc123', 'mem_def456', ...]  — chunks deleted on the next re-index if the file changes
> ```

> [!NOTE]
>
> **Corrupt state is logged** as of PR [#6111](https://github.com/MervinPraison/PraisonAI/pull/6111). If the state file is unreadable (truncated, invalid JSON, permissions), `FileTracker.load()` logs a `WARNING` naming the file and the error, then resets to an empty state. The next `knowledge.index(...)` re-indexes everything — the warning shows the full re-embed came from state loss, not a corpus change.

***

## Common Patterns

### Check success and errors together

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge

k = Knowledge()
result = k.index("./docs", memory={"user_id": "alice"})

if not result.success:
    # One or more files failed to embed — do not treat the corpus as ready.
    for err in result.errors:
        print(f"failed: {err}")
    raise SystemExit(1)

print(f"indexed {result.files_indexed} file(s), {result.chunks_created} chunk(s)")
```

### Check and report results

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge, IndexResult

knowledge = Knowledge()
result = knowledge.index("./docs", memory={"user_id": "alice"})

if not result.success:
    for err in result.errors:
        print(f"Error: {err}")

print(f"Total: {result.files_indexed + result.files_skipped} files")
print(f"New/updated: {result.files_indexed}")
print(f"Unchanged: {result.files_skipped}")
```

### Force full re-index

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./docs",
    memory={"user_id": "alice"},
    force=True,
)
print(f"Full re-index complete: {result.files_indexed} files")
```

### Selective indexing with glob patterns

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge

knowledge = Knowledge()

result = knowledge.index(
    "./project",
    memory={"user_id": "alice"},
    include_glob=["*.md", "*.txt", "*.py"],
    exclude_glob=["*.log", "test_*", "__pycache__/*"],
)
```

### Corpus statistics

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Knowledge, CorpusStats

stats = CorpusStats.from_directory("./docs")
print(f"Files: {stats.file_count}")
print(f"Estimated tokens: {stats.total_tokens}")
print(f"Recommended strategy: {stats.strategy_recommendation}")
```

### Low-level file tracking

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import FileTracker

tracker = FileTracker(state_file=".praison/.index_state.json")
tracker.load()

if tracker.has_changed("./docs/readme.md"):
    print("File needs re-indexing")
```

***

## Best Practices

> [accordion] Use .praisonignore to exclude noisy files
>
> Create a `.praisonignore` in your corpus directory — it follows gitignore syntax and is auto-detected:
>
> ```gitignore theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> *.log
> *.tmp
> test/
> tests/
> __pycache__/
> .env
> secrets.txt
> ```
>
> `.gitignore` is also read as a fallback.

> [accordion] Scope indexes per user for multi-tenant apps
>
> Pass `user_id` or `agent_id` to isolate indexes per tenant. Without a scope identifier, the mem0 backend raises `ScopeRequiredError`.
>
> ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
> result = knowledge.index("./shared-docs", memory={"user_id": user_id})
> ```

> [accordion] Inspect strategy\_recommendation for large corpora
>
> `CorpusStats.strategy_recommendation` returns the optimal retrieval strategy based on file count:
>
> | Files | Strategy |
> | - | - |
> | \< 10 | `direct` |
> | \< 100 | `basic` |
> | \< 1000 | `hybrid` |
> | \< 10000 | `reranked` |
> | \< 100000 | `compressed` |
> | 100000+ | `hierarchical` |

> [accordion] Schedule periodic full re-indexes
>
> Weekly `force=True` re-indexes are still a good hygiene practice for hash-collision paranoia and to recover any IDs that repeatedly fail to delete, but they are no longer required to keep the vector store from growing unbounded on edited files — a changed file's stale chunks are now deleted before its new chunks are added.

***

## Related

> [card] [Knowledge Indexing Errors](https://praison.ai/docs/features/knowledge-indexing-errors.md)
>
> See also — how `add()` and `index()` surface per-file failures together

> [card] [Knowledge Backends](https://praison.ai/docs/features/knowledge-backends.md)
>
> Choose and configure the knowledge storage backend

> [card] [Knowledge](https://praison.ai/docs/features/knowledge.md)
>
> Core knowledge retrieval and agent integration


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.