> ## Documentation Index > Fetch the complete documentation index at: https://docs.stateset.com/llms.txt > Use this file to discover all available pages before exploring further. # Checkpoint recovery and evaluation > Resume exact learner state within its original budget, validate retained checkpoints, nominate a selection for fresh test, and inspect paired quality evidence. Keep four outcomes separate: training completed, learner state can resume, a checkpoint performed well on validation, and a candidate passed an independent test protocol. The [studio](/stateset-agents/finetuning-studio) and [agent CLI](/stateset-agents/finetuning-agents) retain different reviews for each. ## Recover exact training state ```bash theme={null} stateset-agents finetune checkpoints RUN_ID --workspace outputs/finetune stateset-agents finetune prepare-resume RUN_ID \ --checkpoint CHECKPOINT_NAME \ --expected-recovery-sha256 REVIEWED_RECOVERY_SHA \ --workspace outputs/finetune ``` Discovery hashes/checks worker-published checkpoints without loading optimizer state. Retain the chosen `checkpoint`, `checkpoint_sha256`, `recovery_sha256`, saved progress, copy size, and remaining budget. Unavailable choices are explicit; model-only exports are not full recovery checkpoints. Preparation creates a **new private child run**, copying the retained dataset and complete checkpoint. It does not alter the parent or launch. Review the returned child ID and its new `plan_sha256`, then use the normal separate `finetune launch` protocol. Recovery requires Linux, one local training process, a terminal parent with stopped worker and supervisor, identical framework code, a pinned nonzero full model commit, complete learner state, and a worker-published plan-bound receipt. SFT restores its exact trainer, optimizer, scheduler, and RNG checkpoint; native RL restores its saved cursor/runtime state. Changed bytes or settings fail instead of starting fresh. The original total epochs/updates stay fixed; an exhausted checkpoint cannot resume. Inspection is bounded to 100 receipts, 4,096 files per checkpoint, and 1 TiB of bytes. Copying needs the checkpoint, dataset, and 64 MiB overhead in addition to launch resources. Remote recovery is provider-specific. The studio's `c` reuses settings for new training; `z` performs this exact recovery flow. ## Run a local paired evaluation The final-artifact path requires a successful local run, a pinned Hub base model, and a safetensors adapter. Baseline, candidate, and pinned independent judge must already be cached. Loading is cache-only with native model code; missing weights fail rather than download or substitute a stub. Create an evaluation configuration using real paths and a cached judge commit: ```json theme={null} { "cases": "/absolute/path/reviewed-test.json", "judge_model": "organization/independent-judge", "judge_revision": "REPLACE_WITH_FULL_40_CHARACTER_COMMIT", "max_new_tokens": 192, "judge_max_new_tokens": 1024, "context_length": 8192, "seed": 42, "timeout_seconds": 3600, "cpu_slots": 2, "memory_gb": 4.0, "scratch_gb": 5.0 } ``` Cases are schema-version-1 JSON with the declared `test` or `validation` split, reviewed provenance, reviewer, matching freshness flag, and support cases containing `id`, `context`, `user_turns`, `policy_facts`, `expected_resolution`, and `must_avoid`. Do not mark synthetic documentation examples as independently reviewed evidence. ```bash theme={null} stateset-agents finetune evaluation-plan RUN_ID --config evaluation.json stateset-agents finetune evaluation-prepare RUN_ID --config evaluation.json \ --expected-plan-sha256 REVIEWED_EVALUATION_SHA stateset-agents finetune evaluation-launch RUN_ID EVALUATION_ID \ --expected-plan-sha256 REVIEWED_EVALUATION_SHA stateset-agents finetune evaluation-status RUN_ID EVALUATION_ID --verify ``` All commands accept `--workspace`. Planning reads and hashes inputs; preparation snapshots them; launch separately submits queued inference. Retain IDs across disconnects and inspect uncertain submission instead of automatically retrying work. Buildr bounds the job's CPU/RAM/scratch/time; those reservations do not reserve a GPU. Baseline, candidate, and judge run sequentially in separate processes. Policies see fixed user turns and supplied facts with greedy decoding. The judge scores each transcript twice in reversed checklist order. Parsing failures or disagreement above 0.2 reject the evaluation. Success requires score at least 0.8 plus every resolution, accuracy, policy, and non-repetition check in both passes, with no token-limit truncation. This is a one-run fixed-dialogue diagnostic, not adaptive-customer or tool-execution testing. Exact overlap checks do not establish semantic independence. Raw cases, transcripts, and judge recordings remain private under the run's evaluation directory. `--verify` rehashes weights and replays recordings without new model calls. Evaluation stops affect only that evaluation and remain cooperative. ## Validate and compare retained checkpoints `finetune evaluation-targets RUN_ID` discovers checkpoint receipt metadata; discovery explicitly reports `bytes_verified: false`. The evaluation review hashes the selected bytes. Named published checkpoints use `split: validation` unless separately nominated for test. Inference eligibility is broader than exact recovery: an exhausted budget or older trainer does not prevent validation if the terminal local parent and artifact checks pass. ```bash theme={null} stateset-agents finetune validation-compare RUN_ID ANCHOR_EVALUATION_ID stateset-agents finetune validation-comparison-status RUN_ID COMPARISON_ID ``` This queues checkpoint hashing and recorded judge replay without inference. Cohorts must match cases/provenance, model/parent, training/evaluation seeds, judge/rubric, generation, decoding, and code protocol. Consistent repeat evaluations count once, not as independent training seeds. Invalid or conflicting evidence prevents a recommendation. Eligible checkpoints improve over the base without any case-level success, policy, truncation, or required-check regression. Ranking uses successful outcomes, judge mean, then earliest saved step. If none qualifies, retain the base recommendation. In the studio, open `j`, choose a completed checkpoint validation, and press `b` for comparison. ## Nominate for a separate fresh test ```bash theme={null} stateset-agents finetune nomination-plan RUN_ID COMPARISON_ID stateset-agents finetune nomination-prepare RUN_ID COMPARISON_ID \ --expected-plan-sha256 REVIEWED_NOMINATION_SHA stateset-agents finetune nominations RUN_ID ``` Nomination retains a byte-bound selection receipt without copying weights or running a model. Configure the next evaluation with the returned `checkpoint` and `nomination_id`, `split: test`, and a separate reviewed test file. A fresh evaluation review and launch are still required. Unnominated named checkpoints remain validation-only. Test IDs and normalized user turns must not overlap retained validation history, including prepared or failed jobs. Unreadable history or excessive entries fails closed. Exact checks and declared reviewer freshness do not certify semantic or external independence. In the studio, `n` on a completed checkpoint recommendation opens nomination review; `h` reopens saved nominations. The fresh-test form leaves cases blank until separately selected. Nomination does not replace the final adapter, alter completion/quality receipts, or approve deployment. ## Record paired quality evidence ```bash theme={null} stateset-agents finetune quality RUN_ID --workspace outputs/finetune stateset-agents finetune quality RUN_ID --evidence paired-reports.json \ --workspace outputs/finetune ``` Evidence import validates paired test reports and binds one candidate's completed local weights to this run. Inspection rehashes this artifact and recomputes recorded comparisons; it runs no model or judge. Hosted pointers are not supported by this local evidence path. `recorded_study_passed` requires at least six paired training seeds, distinct candidate inventories, at least 32 distinct cases per seed, significance and safety gates, matching protocols, pinned model/judge/rubric identity, reviewer provenance, and this run's recorded training seed. One-seed results remain descriptive. Other artifacts, independence, evaluator identity, and actual train/test isolation still require external review. The workflow does not set `quality_gain_verified` or automatically promote an artifact. Source basis: the October 20, 2026 `docs/FINETUNING_TUI.md`, `FINETUNING_AGENTS.md`, and current CLI schema. See [River's separate training and study contracts](/stateset-agents/river-training) for managed checkpoints and native RL evidence. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.