Skip to main content
Generate an instruction dataset from a teacher LLM, merge parallel batches, then quality-check it before fine-tuning — three commands that compose into a clean training corpus. generate calls an OpenAI-compatible endpoint to synthesise {instruction, input, output} rows from a recipe; dedup merges parallel batches across files with one shared index; validate filters them with research-backed QC checks. All three are YAML-driven, like praisonai-train llm.

​
Quick Start

1

Generate a dataset

One command turns a recipe into a JSONL dataset. Set your teacher endpoint via env vars first.
2

Validate and filter

Quality-check the raw dataset and write the kept rows to a clean file.
3

Fine-tune on the clean dataset

Feed the clean JSONL straight into the trainer.
See praisonai-train Package for the full llm reference.

​
How It Works

generate fans a recipe’s diversity axes into teacher prompts, streams unique rows to JSONL, and validate scores each row against pluggable QC checks.

​
generate

Synthesise {instruction, input, output} rows from a recipe, streamed to a JSONL file with dedup, snapshots, and a stop-file circuit-breaker.

​
CLI flags

​
YAML-only keys

These have no CLI flag — set them in the config file.

​
Behaviour worth knowing

  • output + num_examples are validated up-front — missing either exits 1 with error: 'output' and 'num_examples' (or --num) are required.
  • The output file is only truncated after the first row arrives — a run that fails on credentials, recipe, or the first request never wipes an existing file, and a self-referential dedup_from is read before it would be emptied.
  • Dedup is on-write and cross-run — the normaliser hashes instruction; rows whose normalised instruction is already seen (from dedup_from or earlier in the run) are silently skipped.
  • Zero unique rows exits 1 with error: no rows generated — check endpoint/api_key/deployment and provider JSON-mode support.
  • Progress bar: when tqdm is importable the CLI shows a live bar (generating: 42/1000 [00:12<04:38] req kept=39); otherwise it prints ...500/1000 requests (487 kept) every 500 requests. tqdm stays optional — the import is guarded.
  • Per-row provenance — each emitted row carries model (the teacher deployment), plus task_type and topic when the recipe stamps them (built-in tamil and any _AxisRecipe subclass do). Downstream QC and category splits key off this metadata instead of brittle text detection. Backward compatible — keys are added only when values are available (see Per-row provenance).

​
Example config

Mirror setup/generate.yaml.

​
Python API — generate_dataset()

Import and drive the generator yourself.
Full signature:
Returns an Iterator[dict] of {"instruction", "input", "output"} — plus model, task_type, and topic when available (see Per-row provenance). The caller drives the run by iterating.
progress_callback guarantees (verified by the SDK tests): done is monotonic non-decreasing, kept is monotonic non-decreasing, kept <= done always, final done == total, and final kept == len(rows). When duplicates collapse everything, done still reaches total but kept plateaus. Keep the callback cheap — it runs on the hot path. progress_callback=None preserves the original behaviour exactly.

​
Continuation guard

Halt a run from inside the loop when a billing or quota check flips — a belt-and-suspenders companion to the external stop-file circuit-breaker. should_continue is a fn() -> bool re-checked every sponsor_check_rows completed requests. When it returns False, the run halts via the same clean shutdown as the stop-file: touch the stop-file → cancel pending futures → break. The guard is resolved in a fixed order. Key behaviours:
  • Fail-OPEN — azure_sponsorship_guard returns True on any transient error (az CLI missing, network blip, unparseable output). It never halts a healthy run on a flaky check. Pair it with an external watchdog for hard budget enforcement.
  • Trigger row is emitted, not dropped — the just-completed row is yielded before the guard runs, so the paid, already-finished request that trips the check is never silently lost.
  • sponsor_check_rows coercion — 0 (divide-by-zero) and negative (never-poll) values are corrected to 1; None keeps the 5000 default.
Pass any billing or quota check as a custom callback.
Auto-wire the built-in Azure sponsorship guard with just a subscription id — no callback needed.
Or build the guard directly and pass it as should_continue.
azure_sponsorship_guard(subscription_id, quota_id="Sponsored_2016-01-01") returns a fn() -> bool that is True while the subscription still reads as sponsored, and fail-OPEN on any error.

​
Per-row provenance

Emitted rows are self-describing: the generator stamps model (from deployment) and passes task_type / topic through from the recipe when present. Any axis recipe (tamil, CustomRecipe, or any _AxisRecipe subclass) stamps task_type and topic automatically, so QC and category splits key off metadata instead of brittle text detection.
Rows describe themselves — no re-detection needed.

​
Recipes

tamil is the built-in recipe — a Tamil-language instruction recipe crossing five diversity axes (task × topic × style × audience × variant), registered automatically at import. Define one inline (YAML or dict) without writing Python — pass recipe: as a dict with system, template, and axes (the first two axis keys are the task × topic grid). The template receives {task}, {topic}, {style}, {audience}, {variant} at format time.
Register a custom Python recipe to reuse it by name from the CLI.
Once imported anywhere, --recipe spanish (or recipe: spanish in YAML) works from the CLI. Any _AxisRecipe recipe (including tamil and inline CustomRecipe) automatically stamps task_type (its first axis) and topic (its second axis) onto every emitted row — see Per-row provenance. That metadata is what makes the task-aware QC checks and the metrics.translation by-direction breakdown reliable: they resolve direction from authoritative labels rather than brittle text detection. For the Tamil recipe, the EN_TA_TASK axis value becomes the stamped task_type on translation rows.

​
validate

Quality-check an instruction dataset — dedup, boilerplate/refusal, script purity, truncation, restates-question — plus dataset-level diversity metrics. Thresholds mirror Self-Instruct / LIMA / AlpaGasus / Deita defaults.

​
CLI flags

​
YAML-only keys

​
Checks

Drop checks remove a row; flag checks keep it but count it.

​
Report

validate prints a text report to stdout.
metrics.translation.share is the fraction of kept rows that are translation tasks; by_direction counts them by direction (en_ta, ta_en, unknown); exempted is the number of English-target rows the task-aware exemption actually rescued from the low_script_purity drop. Because both the exemption and the metric use the same detector (translation_intent), the numbers always reconcile. Malformed JSONL lines are skipped with a ⚠ skipping malformed JSON on line N warning — the run still succeeds on the remaining rows.

​
Example config

Mirror setup/validate.yaml.

​
Task-aware script purity

A naive script-purity filter deletes legitimate translation outputs written in the target script — a Tamil→English row’s correct output is English, so a Tamil-purity check would wrongly drop it. Only a resolved English target exempts a row from the Tamil-purity drop. en→ta, ambiguous, and native rows keep the strict thresholds. The detector resolves intent in a fixed order: ground-truth task_type metadata is authoritative → instruction-text keyword fallback for external rows → an ambiguous direction yields no exemption (the safe default). That task_type metadata is now stamped automatically by any axis recipe (not just the Tamil recipe’s EN_TA_TASK task), so freshly-generated corpora resolve direction from authoritative labels out of the box.
task_aware_purity: false restores strict purity behaviour (English translation outputs are dropped again). verify_translation_target: false disables the wrong_target_script flag while leaving the exemption on.

​
Translation composition metric

metrics.translation reports {share, by_direction, exempted} — the corpus-level view of how much of the kept dataset is translation. The metric uses the same translation_intent() detector as the per-row exemption, so the numbers can never disagree with the checks they describe.

​
Python API — score() / filter_rows()

Run the same QC from Python.
Diversity warnings fire when distinct_2 < 0.5, length_cv < 0.4, or top_prefix_share > 0.01 — the “healthy dataset” bar. translation_intent() is the shared detector behind both task-aware purity and the translation metric — call it directly to inspect a single row’s intent.

​
Custom row checks

Register a RowCheck to add a QC rule — no core change.
Once imported, it participates in the next score() / validate run automatically.

​
Cross-file dedup

Merge many JSONL batch files into one deduped file with a single shared MinHash+LSH index — the cross-file duplicates that per-file validate (whose dedup state resets each call) always misses. You generated N batches in parallel with disjoint --start-offset values. cat-ing them and re-running validate still leaves near-duplicates across files. dedup catches them because one index spans every file.

​
CLI flags

​
YAML-only keys

​
Behaviour worth knowing

  • Atomic write — the command writes to a sibling temp file and does os.replace() on success, so an --out that aliases an input is never truncated before its rows are read, and a malformed line never leaves a partial file.
  • Exact-dup key = sha1 of the normalised instruction + input + output, removed first; near-dup runs on the survivors.
  • Missing path raises FileNotFoundError: dedup source not found: PATH — a clear error, not a cryptic AttributeError.
  • Report prints at the end:

​
Which engine to pick

dedup defaults to minhash; validate/score default to sliding. Pick by scale and intent.

​
Example config

​
Python API — global_dedup()

Deduplicate rows across many JSONL paths or in-memory row lists with one shared exact + LSH index — streaming unique rows in source order.
Full signature:
Missing paths raise FileNotFoundError: dedup source not found: PATH.

​
Python API — near_dedup() and MinHashLSH

Drop near-duplicate rows by instruction over either engine — this is what validate’s near-dup pass now delegates to.
Full signature:
Reuse the same index to dedup across a stream of batches — the whole corpus becomes the look-back.
MinHashLSH(threshold=0.7, num_perm=128, n=4, seed=1) is the pure-stdlib index behind both (no datasketch / numpy).
near_dup_method defaults to "sliding" in score/validate but "minhash" in global_dedup/dedup. Existing validate configs are unchanged unless you opt in. An unknown method raises ValueError("unknown near_dup_method 'X'; use 'sliding' or 'minhash'").

​
Common Patterns

​
Disjoint parallel workers

Split a large run across two shells with disjoint --start-offset values, then dedup on both output files.

​
Merge parallel batches

Two shells generate disjoint slices, then one dedup command merges and cross-dedups them before validate.

​
Live progress in a notebook

Pass a progress_callback to drive a UI in Jupyter or Streamlit.

​
Circuit-breaker

Halt a rogue run without losing kept rows — queued requests cancel, in-flight ones drain. Two routes trip the same clean shutdown:
  • External / manual — touch the stop-file from another shell.
  • In-loop / automatic — a should_continue guard (or an auto-wired Azure subscription id) halts the run when a billing or quota check flips, without you touching anything. See Continuation guard.

​
Custom recipe for another language

Point recipe: at an inline dict — no Python needed. See the Spanish example under Recipes.

​
Audit a translation corpus

Read metrics.translation before fine-tuning to verify how much of the corpus is translation and in which direction.

​
Best Practices

List prior output files under dedup_from. Their normalised instructions load into the seen set before the run starts, so a re-run only pays for genuinely new rows.
--snapshot-every 1000 copies the output to snapshots/{stem}_{n}.jsonl every N rows. A crash then costs at most N rows, not the whole run.
The default sliding pass is O(n·near_dup_window) and structurally blind to near-dups more than near_dup_window rows apart. The MinHash + LSH engine has no bounded look-back window and scales sub-linearly. --no-near-dup is still available for exact-dedup-only runs.
Running praisonai-train generate with disjoint --start-offset values still produces near-duplicates across files. praisonai-train dedup batch_*.jsonl --out merged.jsonl removes both exact and near cross-file dups with a shared MinHash+LSH index; cat-ing the files then re-running validate cannot (its dedup state is rebuilt per call).
The callback runs on the hot path after every completed request. Throttle any printing or UI updates inside it so it never becomes the bottleneck.
The defaults target the Tamil block ([2944, 3071]). Set script_range to your language’s Unicode block so low_script_purity and script_review measure the right script.
Any _AxisRecipe recipe now stamps task_type automatically — not just Tamil translation rows — so the detector resolves direction from authoritative metadata instead of the instruction-text fallback. When you build a recipe outside the axis base class, stamp task_type yourself to keep this signal.
The default is 5000. On a fast teacher endpoint that can still be many minutes and dollars of overrun before the guard fires. Match the interval to your acceptable overrun — lower for expensive endpoints.
azure_sponsorship_guard fails OPEN on transient errors (missing az CLI, network blip), so it never halts a healthy run on a flaky check — but that also means it can miss a real flip during an outage. Run an external watchdog too for hard budget enforcement.
Every axis-recipe row is stamped with task_type and topic, so downstream evaluation and category splits can group by these metadata fields directly — no text-heuristic classifier needed.
It surfaces wrong_target_script — failed translations whose output is still in the source script. A naive purity filter keeps these silently because the wrong-script output still passes a same-script purity check.

praisonai-train Package

The standalone training package and its subcommands.

Train CLI

Full flag reference for every train subcommand.

Multi-GPU Training

Fine-tune the clean dataset across multiple GPUs.

Checkpointing

Save, resume, and keep the best checkpoint.

Speed Benchmark

Rank deployments by generation speed — it reuses the same HTTP path as generate.