Docs Code Validation

Automatically validate Python code blocks in documentation

Docs Code Validation

The praisonai docs command extracts and validates Python code blocks from your Mintlify documentation, ensuring all examples are runnable and correct.

Quick Start

# List all Python code blocks in docs
praisonai docs list

# Run all runnable Python blocks (dry-run first)
praisonai docs run --dry-run

# Execute with reports
praisonai docs run --report-dir ./reports

Commands

praisonai docs run

Extracts Python code blocks from documentation, classifies them as runnable or partial, executes runnable blocks, and generates reports.

praisonai docs run [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory
--group, -g	None	Run only specific groups (top-level dirs), repeatable
--include, -i	*	Include patterns (glob)
--exclude, -e	None	Exclude patterns (glob)
--languages, -l	python	Languages to execute (comma-separated)
--timeout, -t	60	Per-snippet timeout in seconds
--max-snippets	None	Maximum snippets to process
--fail-fast, -x	False	Stop on first failure
--dry-run	False	Discover and classify blocks without executing them (safe preview)
--mode, -m	lenient	lenient or strict
--report-dir, -r	./reports/docs/[timestamp]	Report output directory
--no-json	False	Skip JSON report
--no-md	False	Skip Markdown report
--no-csv	False	Skip CSV report
--require-env	None	Required env vars (skip if missing)
--python	Current interpreter	Python executable to use
--quiet, -q	False	Minimal output

Examples:

# Run with custom docs path
praisonai docs run --docs-path /path/to/docs

# Run only specific groups (top-level directories)
praisonai docs run --group models --group tools

# Run only quickstart docs
praisonai docs run --include "quickstart*"

# Run with 2-minute timeout per snippet
praisonai docs run --timeout 120

# Strict mode - fail if any blocks can't be run
praisonai docs run --mode strict

# Limit to first 10 snippets for quick check
praisonai docs run --max-snippets 10

# Use specific Python interpreter
praisonai docs run --python /path/to/python

--dry-run preview mode

Use --dry-run to see exactly what docs run would execute — no subprocesses, no API calls, no cost.

praisonai docs run --dry-run

Every discovered runnable block is written to the report with:

• status: not_run
• skip_reason: "Dry run"
• The runnable_decision field the classifier assigned (so you can audit which blocks would have run)

Dry-run does not validate that code executes correctly — it only proves that the runner sees the block and classifies it as runnable. Use it as a fast preview before a real run, or in PRs to confirm no new blocks were unintentionally added or removed.

When to use --dry-run

Choose between --dry-run, --mode strict, and the default lenient run:

Contributor opening a docs PR — add an .mdx file with a Python block, run praisonai docs run --dry-run --docs-path ./docs, confirm the new block is classified not_run with skip_reason: "Dry run", then push so CI runs the full suite.

Maintainer auditing a large docs tree — run praisonai docs run --dry-run --report-dir ./audit, then filter report.csv for status=not_run to see the full inventory of runnable blocks — no cost, no LLM waiting.

CI PR-preview job — on PR events, call praisonai docs run --dry-run and upload report.md as an artifact so reviewers see what a real run would attempt without spending API tokens.

Example: preview before a real run

# 1. See what would run
praisonai docs run --dry-run --report-dir ./preview

# 2. Inspect
cat ./preview/report.json | jq '.results[] | select(.skip_reason=="Dry run") | {source_path, block_index, runnable_decision}'

# 3. Real run once satisfied
praisonai docs run --report-dir ./actual

praisonai docs list

Lists all discovered code blocks without executing them.

praisonai docs list [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory
--group, -g	None	Filter by group (top-level dir)
--include, -i	None	Include patterns (glob)
--exclude, -e	None	Exclude patterns (glob)
--languages, -l	python	Languages to show
--code, -c	False	Show code preview
--groups	False	Show available groups only

Examples:

# List available groups
praisonai docs list --groups

# List blocks in specific group
praisonai docs list --group models --code

Code Block Classification

The system classifies each Python code block as:

Status	Description
Runnable	Has imports AND terminal action (.start(), print(), etc.)
Partial	Missing imports or incomplete - not executed
Skipped	Marked with skip directive or contains input()

Terminal Actions Detected

• agents.start() / agent.start()
• print()
• asyncio.run()
• if __name__ == "__main__":

Directives

Control execution behavior with HTML comments above code blocks:

<!-- praisonai: runnable=true timeout=120 require_env=OPENAI_API_KEY -->
```python
from praisonaiagents import Agent
agent = Agent(instructions="test")
agent.start()

### Available Directives

| Directive | Values | Description |
|-----------|--------|-------------|
| `runnable` | `true/false` | Force block to be runnable or skipped |
| `skip` | `true/false` | Skip this block entirely |
| `timeout` | seconds | Custom timeout for this block |
| `require_env` | `KEY1,KEY2` | Required environment variables |

## Reports

Reports are generated in the specified directory:

reports/docs/[timestamp]/ ├── report.json # Machine-readable results ├── report.md # Human-readable summary ├── report.csv # CSV for quick scanning/filtering └── logs/ ├── quickstart__0.stdout.log └── quickstart__0.stderr.log

### CSV Report

The CSV report contains one row per item for easy filtering:

| Column | Description |
|--------|-------------|
| `suite` | "docs" |
| `group` | Top-level directory (e.g., "models", "tools") |
| `item_id` | Unique identifier |
| `source_path` | Path to doc file |
| `block_index` | Code block index in file |
| `status` | passed/failed/skipped/timeout/not_run |
| `duration_seconds` | Execution time |
| `error_message` | Error details if failed |
| `code_hash` | Content hash for change tracking |

### JSON Report Schema

```json
{
  "metadata": {
    "timestamp": "2026-01-09T11:30:00Z",
    "platform": "Darwin-23.0.0-arm64",
    "python_version": "3.11.0",
    "praisonai_version": "2.0.0",
    "docs_path": "/path/to/docs",
    "totals": {
      "passed": 10,
      "failed": 2,
      "skipped": 5,
      "timeout": 0,
      "not_run": 15
    }
  },
  "results": [
    {
      "item_id": "docs/quickstart.mdx::0",
      "source_path": "docs/quickstart.mdx",
      "block_index": 0,
      "language": "python",
      "line_start": 30,
      "line_end": 39,
      "runnable_decision": "heuristic_standalone",
      "status": "not_run",
      "skip_reason": "Dry run",
      "code_hash": "sha256:…"
    }
  ]
}

skip_reason: "Dry run" identifies items that were only previewed. Other not_run items come from runnable=false directives or lenient-mode classification.

CI/CD Integration

Add to your GitHub Actions workflow:

name: Docs Validation

on: [push, pull_request]

jobs:
  validate-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install praisonai
      
      - name: Validate docs code blocks
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          praisonai docs run \
            --docs-path ./docs \
            --report-dir ./reports \
            --no-stream
      
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: docs-validation-report
          path: ./reports/

Pull-request preview (no API keys required):

- name: Preview docs code blocks (no execution)
  run: |
    praisonai docs run \
      --docs-path ./docs \
      --dry-run \
      --report-dir ./preview

- name: Comment discovery summary
  # Post the report.md as a PR comment so reviewers see what a full run would attempt

praisonai docs run-all

Runs all documentation code blocks group-by-group with parallel execution by default.

praisonai docs run-all [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory
--timeout, -t	60	Per-snippet timeout in seconds
--report-dir, -r	~/Downloads/reports/docs/[timestamp]	Report output directory
--parallel/--sequential	parallel	Run groups in parallel or sequential
--workers, -w	4	Max parallel workers
--quiet, -q	False	Minimal output
--ci	False	CI-friendly output (no colors, proper exit codes)

Examples:

# Run all groups in parallel (default)
praisonai docs run-all

# Run sequentially for debugging
praisonai docs run-all --sequential

# Run with more workers for faster execution
praisonai docs run-all --workers 8

# CI-friendly output
praisonai docs run-all --ci

praisonai docs stats

Shows statistics for documentation code blocks by group.

praisonai docs stats [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory
--group, -g	None	Filter by group (top-level dir)
--languages, -l	python	Languages to show

Exit Codes

Code	Meaning
0	All runnable blocks passed
1	One or more blocks failed or timed out
2	Invalid path or configuration error

In strict mode (--mode strict), exit code 1 is also returned if any blocks are marked as not_run.

---

CLI Command Validation

The praisonai docs cli subcommand group validates CLI commands found in bash code blocks within documentation. It discovers praisonai commands and validates them by running with --help to ensure they exist and work correctly.

Quick Start

# Show CLI command statistics
praisonai docs cli stats

# List all discovered CLI commands
praisonai docs cli list

# Validate all CLI commands from docs
praisonai docs cli run-all

# Validate with limited scope
praisonai docs cli run-all --max-items 10 --timeout 5

praisonai docs cli run-all

Validates all CLI commands from documentation by running each with --help.

praisonai docs cli run-all [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	~/PraisonAIDocs/docs/cli	Path to CLI documentation directory
--timeout, -t	10	Per-command timeout in seconds
--report-dir, -r	~/Downloads/reports/docs-cli/<timestamp>	Report output directory
--parallel/--sequential	parallel	Run commands in parallel or sequential
--workers, -w	4	Max parallel workers
--max-items	None	Maximum commands to run
--group, -g	None	Run only specific groups (repeatable)
--quiet, -q	False	Minimal output
--ci	False	CI-friendly output (no colors, proper exit codes)

Examples:

# Run all CLI commands in parallel (default)
praisonai docs cli run-all

# Validate first 10 commands for quick check
praisonai docs cli run-all --max-items 10

# Run with more workers for faster execution
praisonai docs cli run-all --workers 8

# CI-friendly output with proper exit codes
praisonai docs cli run-all --ci --timeout 5

# Run sequentially for debugging
praisonai docs cli run-all --sequential

praisonai docs cli list

Lists all discovered CLI commands from documentation without executing them.

praisonai docs cli list [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory
--group, -g	None	Filter by group (repeatable)
--runnable	False	Show only runnable commands
--groups	False	Show available groups only

Examples:

# List all CLI commands
praisonai docs cli list

# List available groups
praisonai docs cli list --groups

# List only runnable commands
praisonai docs cli list --runnable

praisonai docs cli stats

Shows statistics for CLI commands found in documentation.

praisonai docs cli stats [OPTIONS]

Options:

Option	Default	Description
--docs-path, -p	Auto-detect	Path to documentation directory

Example Output:

CLI Commands Statistics
========================================
Total commands:    1463
Runnable:          933
Skipped:           530

By Group:
----------------------------------------
  databases              31/37   runnable
  monitoring              8/9    runnable
  root                  894/1417 runnable

praisonai docs cli report

View CLI validation report with failures and error grouping.

praisonai docs cli report [REPORT_DIR] [OPTIONS]

Options:

Option	Default	Description
--base-dir, -b	~/Downloads/reports/docs-cli	Base directory for auto-detection
--limit, -n	20	Max items to show (0 = unlimited)
--format, -f	table	Output format: table or json

Examples:

# View latest report
praisonai docs cli report

# View specific report directory
praisonai docs cli report ./my-report

# JSON output for parsing
praisonai docs cli report --format json

Command Classification

CLI commands are classified as:

Status	Description
Runnable	Valid praisonai command without placeholders
Skipped	Contains placeholders like <file>, [OPTIONS], or path/to/

Placeholder Detection

Commands are automatically skipped if they contain:

• [OPTIONS] or [...]
• <file>, <name>, <path> style placeholders
• path/to/, /my/project/ style example paths
• your_, example_ prefixed values

CI/CD Integration for CLI Validation

Add to your GitHub Actions workflow:

name: CLI Docs Validation

on: [push, pull_request]

jobs:
  validate-cli-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install praisonai
      
      - name: Validate CLI commands in docs
        run: |
          praisonai docs cli run-all \
            --docs-path ./docs/cli \
            --ci \
            --timeout 10
      
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: cli-validation-report
          path: ~/Downloads/reports/docs-cli/