Skip to main content
Generate an instruction dataset from a teacher LLM, merge parallel batches, then quality-check it before fine-tuning — three commands that compose into a clean training corpus. generate calls an OpenAI-compatible endpoint to synthesise {instruction, input, output} rows from a recipe; dedup merges parallel batches across files with one shared index; validate filters them with research-backed QC checks. All three are YAML-driven, like praisonai-train llm.

​
Quick Start

1

Generate a dataset

One command turns a recipe into a JSONL dataset. Set your teacher endpoint via env vars first.
2

Validate and filter

Quality-check the raw dataset and write the kept rows to a clean file.
3

Fine-tune on the clean dataset

Feed the clean JSONL straight into the trainer.
See praisonai-train Package for the full llm reference.

​
How It Works

generate fans a recipe’s diversity axes into teacher prompts, streams unique rows to JSONL, and validate scores each row against pluggable QC checks.

​
generate

Synthesise {instruction, input, output} rows from a recipe, streamed to a JSONL file with dedup, snapshots, and a stop-file circuit-breaker.

​
CLI flags

​
YAML-only keys

These have no CLI flag — set them in the config file.

​
Behaviour worth knowing

  • output + num_examples are validated up-front — missing either exits 1 with error: 'output' and 'num_examples' (or --num) are required.
  • Atomic write — generate writes to a sibling temp file (via tempfile.mkstemp next to the destination) and os.replaces the destination only when at least one row is produced. A run that fails on credentials, recipe, or every teacher request (no JSON-mode support, all timeouts, etc.) leaves any pre-existing file byte-for-byte untouched — the temp file is unlinked and the CLI still exits 1 with error: no rows generated. A self-referential dedup_from is also read before it would be emptied. Fixed in PraisonAI PR #4239.
  • File permissions preserved on overwrite — when the destination already exists, the atomic swap keeps its existing mode. For a fresh file, mode falls back to 0o666 & ~umask (what a plain open(path, "w") would have produced), so downstream validation / training jobs can still read a shared corpus.
  • Symlink destinations are followed — --output pointing at a symlink updates the file the link points at, rather than replacing the link itself with a regular file. Consumers reading through the link continue to see the fresh corpus.
  • Dedup is on-write and cross-run — the normaliser hashes instruction; rows whose normalised instruction is already seen (from dedup_from or earlier in the run) are silently skipped.
  • Zero unique rows exits 1 with error: no rows generated — check endpoint/api_key/deployment and provider JSON-mode support.
  • Progress bar: when tqdm is importable the CLI shows a live bar (generating: 42/1000 [00:12<04:38] req kept=39); otherwise it prints ...500/1000 requests (487 kept) every 500 requests. tqdm stays optional — the import is guarded.
  • Per-row provenance — each emitted row carries model (the teacher deployment), plus task_type and topic when the recipe stamps them (built-in tamil and any _AxisRecipe subclass do). Downstream QC and category splits key off this metadata instead of brittle text detection. Backward compatible — keys are added only when values are available (see Per-row provenance).

​
Example config

Mirror setup/generate.yaml.

​
Python API — generate_dataset()

Import and drive the generator yourself.
Full signature:
Returns an Iterator[dict] of {"instruction", "input", "output"} — plus model, task_type, and topic when available (see Per-row provenance). The caller drives the run by iterating.
progress_callback guarantees (verified by the SDK tests): done is monotonic non-decreasing, kept is monotonic non-decreasing, kept <= done always, final done == total, and final kept == len(rows). When duplicates collapse everything, done still reaches total but kept plateaus. Keep the callback cheap — it runs on the hot path. progress_callback=None preserves the original behaviour exactly.

​
Continuation guard

Halt a run from inside the loop when a billing or quota check flips — a belt-and-suspenders companion to the external stop-file circuit-breaker. should_continue is a fn() -> bool re-checked every sponsor_check_rows completed requests. When it returns False, the run halts via the same clean shutdown as the stop-file: touch the stop-file → cancel pending futures → break. The guard is resolved in a fixed order. Key behaviours:
  • Fail-OPEN — azure_sponsorship_guard returns True on any transient error (az CLI missing, network blip, unparseable output). It never halts a healthy run on a flaky check. Pair it with an external watchdog for hard budget enforcement.
  • Trigger row is emitted, not dropped — the just-completed row is yielded before the guard runs, so the paid, already-finished request that trips the check is never silently lost.
  • sponsor_check_rows coercion — 0 (divide-by-zero) and negative (never-poll) values are corrected to 1; None keeps the 5000 default.
Pass any billing or quota check as a custom callback.
Auto-wire the built-in Azure sponsorship guard with just a subscription id — no callback needed.
Or build the guard directly and pass it as should_continue.
azure_sponsorship_guard(subscription_id, quota_id="Sponsored_2016-01-01") returns a fn() -> bool that is True while the subscription still reads as sponsored, and fail-OPEN on any error.

​
Per-row provenance

Emitted rows are self-describing: the generator stamps model (from deployment) and passes task_type / topic through from the recipe when present. Any axis recipe (tamil, CustomRecipe, or any _AxisRecipe subclass) stamps task_type and topic automatically, so QC and category splits key off metadata instead of brittle text detection.
Rows describe themselves — no re-detection needed.

​
Recipes

tamil is the built-in recipe — a Tamil-language instruction recipe crossing five diversity axes (task × topic × style × audience × variant), registered automatically at import. Define one inline (YAML or dict) without writing Python — pass recipe: as a dict with system, template, and axes (the first two axis keys are the task × topic grid). The template receives {task}, {topic}, {style}, {audience}, {variant} at format time.
Register a custom Python recipe to reuse it by name from the CLI.
Once imported anywhere, --recipe spanish (or recipe: spanish in YAML) works from the CLI. Any _AxisRecipe recipe (including tamil and inline CustomRecipe) automatically stamps task_type (its first axis) and topic (its second axis) onto every emitted row — see Per-row provenance. That metadata is what makes the task-aware QC checks and the metrics.translation by-direction breakdown reliable: they resolve direction from authoritative labels rather than brittle text detection. For the Tamil recipe, the EN_TA_TASK axis value becomes the stamped task_type on translation rows.

​
validate

Quality-check an instruction dataset — dedup, boilerplate/refusal, script purity, truncation, restates-question — plus dataset-level diversity metrics. Thresholds mirror Self-Instruct / LIMA / AlpaGasus / Deita defaults.
Script purity defaults to the Tamil block. validate drops outputs that fall below a purity floor for a target Unicode block, and that block defaults to Tamil (script_range: [2944, 3071] # U+0B80–U+0BFF). For any other language, set script_range in your config (e.g. [65, 591] for Latin) — otherwise non-Tamil outputs are dropped as low_script_purity and the run exits 1.

​
CLI flags

​
YAML-only keys

​
Checks

Drop checks remove a row; flag checks keep it but count it.

​
Report

validate prints a text report to stdout.
metrics.translation.share is the fraction of kept rows that are translation tasks; by_direction counts them by direction (en_ta, ta_en, unknown); exempted is the number of English-target rows the task-aware exemption actually rescued from the low_script_purity drop. Because both the exemption and the metric use the same detector (translation_intent), the numbers always reconcile. Malformed JSONL lines are skipped with a ⚠ skipping malformed JSON on line N warning — the run still succeeds on the remaining rows.

​
Behaviour worth knowing

  • Atomic write — validate writes to a sibling temp file next to the destination and os.replaces it only when at least one row survived QC. An --out that aliases the input is never truncated when everything is dropped; the existing corpus stays byte-for-byte intact. File mode and symlink destinations are handled the same as generate/dedup.
  • Zero-kept exits 1 — when QC drops every row, validate exits 1 with error: no rows kept — all rows were dropped by QC ({drops}); any existing --out was left intact. The drops dict names the actual reasons — typically low_script_purity for wrong-language corpora. Previously the command silently exited 0 and could truncate an aliased --out to zero bytes. Fixed in PraisonAI PR #4316.

​
Example config

Mirror setup/validate.yaml.

​
Task-aware script purity

A naive script-purity filter deletes legitimate translation outputs written in the target script — a Tamil→English row’s correct output is English, so a Tamil-purity check would wrongly drop it. Only a resolved English target exempts a row from the Tamil-purity drop. en→ta, ambiguous, and native rows keep the strict thresholds. The detector resolves intent in a fixed order: ground-truth task_type metadata is authoritative → instruction-text keyword fallback for external rows → an ambiguous direction yields no exemption (the safe default). That task_type metadata is now stamped automatically by any axis recipe (not just the Tamil recipe’s EN_TA_TASK task), so freshly-generated corpora resolve direction from authoritative labels out of the box.
task_aware_purity: false restores strict purity behaviour (English translation outputs are dropped again). verify_translation_target: false disables the wrong_target_script flag while leaving the exemption on.

​
Translation composition metric

metrics.translation reports {share, by_direction, exempted} — the corpus-level view of how much of the kept dataset is translation. The metric uses the same translation_intent() detector as the per-row exemption, so the numbers can never disagree with the checks they describe.

​
Python API — score() / filter_rows()

Run the same QC from Python.
Diversity warnings fire when distinct_2 < 0.5, length_cv < 0.4, or top_prefix_share > 0.01 — the “healthy dataset” bar. translation_intent() is the shared detector behind both task-aware purity and the translation metric — call it directly to inspect a single row’s intent.

​
Custom row checks

Register a RowCheck to add a QC rule — no core change.
Once imported, it participates in the next score() / validate run automatically.

​
from-trials

Export a scored trials report into a trainer-ready SFT dataset — keep the attempts that passed verification, drop the rest, and write a ShareGPT (or Alpaca) JSONL the praisonai-train llm trainer consumes unchanged. from-trials closes the loop from verification to data generation: verify the agent, keep what passed (rejection sampling), fine-tune on it, then re-verify to measure the gain — one supported pipeline instead of a hand-rolled script. It is invoked under the data group as praisonai-train data from-trials REPORT [OPTIONS].

​
Quick Start

1

Export the passing attempts

Take a saved trials report and write the passing, tool-free attempts as a training dataset.
2

Fine-tune on the exported dataset

Feed the JSONL straight into the trainer — no reshaping needed.
3

Re-verify to measure the gain

Re-run the same trials on the fine-tuned model and compare pass rates.

​
CLI flags

​
Behaviour worth knowing

  • Selection discipline — every skip is counted, never silent. Four filters run in order, each with its own counter:
    • score is None → never a candidate (skipped_unscored; kept under --all, which sets only_passed=False).
    • only_passed=True (default) → drop verifier-failed attempts (skipped_failed).
    • frontier_only=True (default) → drop cases where pass_rate == 0 or pass_rate == 1 (skipped_saturated) — saturated cases add K near-duplicates of already-mastered behaviour, zero-pass cases have nothing to export.
    • Tool-using runs excluded by default (skipped_tool_runs) — exporting only the final text of a run that used tools would train the model to answer without the tools it actually needed.
  • Deterministic emission order — case, then attempt, so output files are byte-reproducible.
  • Exit code 1 on zero rows written — with a stderr hint pointing at --all / --include-saturated, or to check that the report has passing tool-free attempts.
  • Alpaca vs ShareGPT — the (often sole) user prompt is never dropped. With a system turn, instruction = system and input = "\n".join(all user turns); without a system turn, instruction = first user turn and input = "\n".join(remaining user turns). Conversations are truncated at the final assistant turn; assistant content is the raw generated text.
  • Pass-threshold handling — resolved in a fixed order: an explicit passed boolean wins; otherwise the attempt’s score is compared against its own threshold (pass_threshold / threshold / min_score), so 0.3 under a pass_threshold: 0.5 counts as a failure; only when no threshold is carried does it fall back to score > 0.

​
Console output

from-trials prints a summary block naming every counter, then the write and provenance paths.

​
Output format

ShareGPT (--format messages, default) — one JSON object per line; the trainer’s process_dataset standardises it directly.
Alpaca (--format alpaca) — flat instruction/input/output rows.

​
Python API — export_trials()

Import the exporter and its summary dataclass from the friendly top-level namespace.
Full signature:
Returns an ExportSummary dataclass — six counter fields plus a to_dict() method and a CLI-friendly __str__ that emits written=X skipped_failed=Y ....

​
Provenance sidecar

Every run writes {path}.jsonl.meta.json alongside the dataset — every emitted row is traceable to its source case, attempt, and score.
only_passed=True is rejection sampling — it amplifies behaviour the agent already produces and inherits any bias in the scorer/judge that decided a run “passed”. It cannot teach behaviour the agent never exhibited; treat it as reinforcing verified wins, not as an oracle.

​
Export trials with QC (from-trials --qc)

Turn passing agent trials into a training corpus and quality-check them in one pass — --qc reuses the validate filter but auto-disables the target-script check, because agent trajectories are English by construction. --qc runs dedup, boilerplate/refusal, length, and diversity — not the Tamil script-purity check. The export QC path sets script_drop/script_flag to 0 so nothing falls below the target-script floor, which is why English trajectories survive instead of being dropped wholesale as low_script_purity. Opt back into a script check for another language by passing qc_cfg to export_trials() with script_range (and/or script_drop / script_flag). Any one of those keys re-enables the target-script check with the QC filter’s own defaults (0.50 / 0.70) — so a documented qc_cfg={"script_range": [...]} no longer silently passes wrong-script rows.
The CLI mirrors this — pass --qc, and when QC drops everything the error names the real reasons instead of guessing.

​
ExportSummary QC fields

export_trials() returns an ExportSummary that now tracks QC removals. str(summary) appends skipped_qc=N, and qc_drops={...} when non-empty.

​
Cross-file dedup

Merge many JSONL batch files into one deduped file with a single shared MinHash+LSH index — the cross-file duplicates that per-file validate (whose dedup state resets each call) always misses. You generated N batches in parallel with disjoint --start-offset values. cat-ing them and re-running validate still leaves near-duplicates across files. dedup catches them because one index spans every file.

​
CLI flags

​
YAML-only keys

​
Behaviour worth knowing

  • Atomic write — the command writes to a sibling temp file and does os.replace() on success, so an --out that aliases an input is never truncated before its rows are read, and a malformed line never leaves a partial file.
  • File mode and symlinks handled the same as generate — the atomic swap preserves the destination’s existing mode (or 0o666 & ~umask for a fresh file), and a symlink destination is followed so the swap lands on the target, not the link itself. See PraisonAI PR #4239.
  • Exact-dup key = sha1 of the normalised instruction + input + output, removed first; near-dup runs on the survivors.
  • Missing path raises FileNotFoundError: dedup source not found: PATH — a clear error, not a cryptic AttributeError.
  • Report prints at the end:

​
Which engine to pick

dedup defaults to minhash; validate/score default to sliding. Pick by scale and intent.

​
Example config

​
Python API — global_dedup()

Deduplicate rows across many JSONL paths or in-memory row lists with one shared exact + LSH index — streaming unique rows in source order.
Full signature:
Missing paths raise FileNotFoundError: dedup source not found: PATH.

​
Python API — near_dedup() and MinHashLSH

Drop near-duplicate rows by instruction over either engine — this is what validate’s near-dup pass now delegates to.
Full signature:
Reuse the same index to dedup across a stream of batches — the whole corpus becomes the look-back.
MinHashLSH(threshold=0.7, num_perm=128, n=4, seed=1) is the pure-stdlib index behind both (no datasketch / numpy).
near_dup_method defaults to "sliding" in score/validate but "minhash" in global_dedup/dedup. Existing validate configs are unchanged unless you opt in. An unknown method raises ValueError("unknown near_dup_method 'X'; use 'sliding' or 'minhash'").

​
Common Patterns

​
Disjoint parallel workers

Split a large run across two shells with disjoint --start-offset values, then dedup on both output files.

​
Merge parallel batches

Two shells generate disjoint slices, then one dedup command merges and cross-dedups them before validate.

​
Live progress in a notebook

Pass a progress_callback to drive a UI in Jupyter or Streamlit.

​
Circuit-breaker

Halt a rogue run without losing kept rows — queued requests cancel, in-flight ones drain. Two routes trip the same clean shutdown:
  • External / manual — touch the stop-file from another shell.
  • In-loop / automatic — a should_continue guard (or an auto-wired Azure subscription id) halts the run when a billing or quota check flips, without you touching anything. See Continuation guard.

​
Custom recipe for another language

Point recipe: at an inline dict — no Python needed. See the Spanish example under Recipes.

​
Audit a translation corpus

Read metrics.translation before fine-tuning to verify how much of the corpus is translation and in which direction.

​
Preference dataset shapes

Preference tuning reads columns by name — give each method the exact shape it needs. See Preference Tuning.
Preference datasets are not flattened to a text column (that’s SFT-only behaviour) — the columns above survive to the trainer. PraisonAI validates them before the run starts and names any that are missing, so a shape mismatch fails in seconds.
dataset.processing_func is not honoured today. Setting it now logs a WARNING and is ignored (it was silently ignored before). Do your row transforms with the rename / filter_* keys or preprocess the file offline.

​
Best Practices

List prior output files under dedup_from. Their normalised instructions load into the seen set before the run starts, so a re-run only pays for genuinely new rows.
--snapshot-every 1000 copies the output to snapshots/{stem}_{n}.jsonl every N rows. A crash then costs at most N rows, not the whole run.
The default sliding pass is O(n·near_dup_window) and structurally blind to near-dups more than near_dup_window rows apart. The MinHash + LSH engine has no bounded look-back window and scales sub-linearly. --no-near-dup is still available for exact-dedup-only runs.
Running praisonai-train generate with disjoint --start-offset values still produces near-duplicates across files. praisonai-train dedup batch_*.jsonl --out merged.jsonl removes both exact and near cross-file dups with a shared MinHash+LSH index; cat-ing the files then re-running validate cannot (its dedup state is rebuilt per call).
The callback runs on the hot path after every completed request. Throttle any printing or UI updates inside it so it never becomes the bottleneck.
The defaults target the Tamil block ([2944, 3071]). Set script_range to your language’s Unicode block so low_script_purity and script_review measure the right script.from-trials --qc auto-disables the target-script check for English agent trajectories, so exports aren’t dropped as low_script_purity out of the box. For other languages, pass qc_cfg={"script_range": [...]} to export_trials() to re-enable the check with the QC defaults (0.50 / 0.70).
Any _AxisRecipe recipe now stamps task_type automatically — not just Tamil translation rows — so the detector resolves direction from authoritative metadata instead of the instruction-text fallback. When you build a recipe outside the axis base class, stamp task_type yourself to keep this signal.
The default is 5000. On a fast teacher endpoint that can still be many minutes and dollars of overrun before the guard fires. Match the interval to your acceptable overrun — lower for expensive endpoints.
azure_sponsorship_guard fails OPEN on transient errors (missing az CLI, network blip), so it never halts a healthy run on a flaky check — but that also means it can miss a real flip during an outage. Run an external watchdog too for hard budget enforcement.
Every axis-recipe row is stamped with task_type and topic, so downstream evaluation and category splits can group by these metadata fields directly — no text-heuristic classifier needed.
It surfaces wrong_target_script — failed translations whose output is still in the source script. A naive purity filter keeps these silently because the wrong-script output still passes a same-script purity check.
generate synthesises rows from a teacher LLM; from-trials mines your own verified agent runs. When you already have a scored trials report, praisonai-train data from-trials keeps only the attempts that passed — reinforcing behaviour you measured rather than behaviour a teacher imagined. Treat it as rejection sampling: it amplifies verified wins, so it inherits the scorer’s bias and cannot teach behaviour the agent never exhibited.

praisonai-train Package

The standalone training package and its subcommands.

Train CLI

Full flag reference for every train subcommand.

Multi-GPU Training

Fine-tune the clean dataset across multiple GPUs.

Checkpointing

Save, resume, and keep the best checkpoint.

Speed Benchmark

Rank deployments by generation speed — it reuses the same HTTP path as generate.

Trials Engine

Run K scored attempts per case, then feed the report into from-trials.