> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Infer (one-shot inference)

> Prompt a model you just trained and see it stream

Prompt a model you just trained and watch it stream a reply — the fastest way to answer "did it work?".

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai-train infer -d lora_model "Summarise this release."
```

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "praisonai-train infer"
        P[💬 prompt] --> Load[📦 load adapter]
        Load --> Tmpl[🧩 chat template]
        Tmpl --> Stream[✅ streamed tokens]
    end

    classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef process fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef output fill:#10B981,stroke:#7C90A0,color:#fff

    class P input
    class Load,Tmpl process
    class Stream output
```

> [!NOTE]
>
> The command is named `infer`, **not** `generate` — `generate` is dataset generation and reusing it would displace that command.

## Quick Start

**1. Prompt the trained model**

Point at a trained adapter and pass a prompt — the reply streams token-by-token.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai-train infer -d lora_model "Summarise this release."
```

**2. Tune the output length and sampling**

Bump `--max-new-tokens` for longer replies; lower `--temperature` for more focused output.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai-train infer -d lora_model "Write release notes." \
    --max-new-tokens 512 --temperature 0.3
```

***

## How It Works

`infer` loads the trained model with Unsloth, applies the tokenizer's chat template with a `user` role, and streams the generation straight to stdout.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant CLI as praisonai-train infer
    participant Unsloth as FastLanguageModel
    participant Tok as Tokenizer
    participant Model

    User->>CLI: infer -d lora_model "prompt"
    CLI->>Unsloth: from_pretrained + for_inference
    Unsloth-->>CLI: model + tokenizer
    CLI->>Tok: apply chat template (role: user)
    Tok-->>CLI: input ids
    CLI->>Model: generate (TextStreamer, skip_prompt=True)
    Model-->>User: tokens stream to stdout
```

The model is loaded via Unsloth's `FastLanguageModel.from_pretrained` + `for_inference`, and generation streams through `TextStreamer(tokenizer, skip_prompt=True)`.

***

## Configuration Options

| Option | Short | Type | Default | Description |
| - | - | - | - | - |
| `prompt` (positional) | — | `str` | *required* | What to send the model. |
| `--model-dir` | `-d` | `Path` (must exist) | `lora_model` | Path to a trained adapter or model. |
| `--max-new-tokens` | — | `int` | `256` | Maximum new tokens to generate. |
| `--temperature` | — | `float` | `0.7` | Sampling temperature. |
| `--max-seq-length` | — | `int` | `2048` | Context length used when loading the model. |
| `--load-in-4bit` / `--no-load-in-4bit` | — | `bool` | `True` | Load the model in 4-bit quantisation. |

> [!NOTE]
>
> If the Unsloth / Transformers stack isn't installed, `infer` prints an error with the remediation `pip install "praisonai-train[llm]"` and exits `1`.

***

## Streaming

The prompt is skipped (`skip_prompt=True`), so only the model's reply prints — the first token is your "is it alive?" signal, arriving long before the last on a large model.

***

## Choosing between infer, serve, benchmark

Three commands touch a trained model — pick by what you're trying to answer.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Q{What do you need?}
    Q -->|Prove the fine-tune works| I[✅ infer<br/>one prompt, streamed reply]
    Q -->|Persistent OpenAI endpoint| S[✅ serve<br/>OpenAI-compatible HTTP]
    Q -->|tokens/sec on a remote API| B[✅ benchmark<br/>rank deployments]

    classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef success fill:#10B981,stroke:#7C90A0,color:#fff

    class Q question
    class I,S,B success
```

`infer` runs a single local generation; `serve` stands up an OpenAI-compatible endpoint; `benchmark` measures throughput on already-deployed remote APIs.

***

## Best Practices

> [accordion] Install the \[llm] extra first
>
> `infer` needs the Unsloth / Transformers stack. Run `pip install "praisonai-train[llm]"` — otherwise the command exits `1` with that exact remediation.

> [accordion] Keep the defaults for a sanity check
>
> `temperature=0.7` and `max-new-tokens=256` are deliberately usable defaults. Leave them for a first "did my fine-tune take?" check and only bump `--max-new-tokens` when you need longer outputs.

> [accordion] Disable 4-bit only when you need to
>
> Turn off `--load-in-4bit` only if the model doesn't fit in 4-bit or you're comparing against full-precision output. The 4-bit default keeps memory low.

> [accordion] One prompt, one generation
>
> `infer` is not a chat loop — it sends a single prompt and streams one reply, then exits. For a persistent endpoint use `serve` instead.

***

## Related

> [card] [Checkpoints CLI](https://praison.ai/docs/features/praisonai-train-checkpoints.md)
>
> Discover what a training run saved without `ls`.

> [card] [Serve & MTP Fast-Inference](https://praison.ai/docs/features/praisonai-train-serve.md)
>
> Serve a GGUF over an OpenAI-compatible endpoint.

> [card] [Export a trained model](https://praison.ai/docs/features/praisonai-train-export.md)
>
> Publish a trained model to HF, GGUF, or Ollama.

> [card] [Train CLI](https://praison.ai/docs/cli/train.md)
>
> Full flag reference for every subcommand.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.