On-device tuning

services/tuning/ is the "automated tunnel" that lets a model you downloaded or ran in IMURUN get better from your own use. You mark answers you like (thumbs-up) or correct the ones that were wrong ("teach"); when there is enough of that, the channel builds a dataset, trains a LoRA adapter on the original weights of that model, checks the result did not get worse, exports a new GGUF, and offers it to you with a Use it button. Everything runs on this machine.

It is off by default and needs a separate ~5 GB training stack that is installed only when you ask for it.

What it can and cannot do (read this first)#

The request that started this was: "I asked about a person and got a made-up answer with a knowledge-cutoff disclaimer; I want the model to get better from use." Be clear about what fine-tuning fixes:

The channel is honest in its outputs too: every run reports held-out loss before and after, and a run that regresses on a fixed sanity set ("The capital of France is" must still produce "Paris") is rejected and not exported.

Feasibility on this machine (RTX 3050 Laptop, 4 GB VRAM; 16 GB RAM)#

The planner (services/tuning/feasibility.py) probes the real hardware (nvidia-smi, RAM, RAM+pagefile, free disk) and picks the first method that fits; the table below is what it decides here. It is enforced in code: impossible is refused, slow needs an explicit allow_slow (CLI --allow-slow) and the response shows the time estimate first.

How the memory figures are computed (batch 1, sequence ≤ 512, gradient checkpointing, r = 16): QLoRA ≈ 0.6 GB per billion params + 0.9 GB; LoRA on a bf16 base ≈ 2.0 GB/B + 0.9 GB; CPU fp32 ≈ 4 GB/B + 2 GB; CPU bf16 ≈ 2 GB/B + 2 GB; CPU 4-bit ≈ 0.6 GB/B + 2 GB. 0.6 GB of VRAM and 3 GB of RAM are kept back for the OS and the desktop. QLoRA needs bitsandbytes; on this machine setup probed it working on both the GPU and the CPU (bitsandbytes 0.50.2 has native Windows wheels). Without bitsandbytes the 1B still fits as plain LoRA on a bf16 base (3.4 / 3.4 GB).

Time estimates start from first guesses (QLoRA GPU ≈ 600 tokens/s for a 1B, scaled by 1/params; the CPU 4-bit rate is the measured one below) and are replaced by the throughput each finished run measured on this machine (~/.imurun/tuning/hardware.json).

Measured on this machine (2026-11-01)#

Real runs, not estimates (tests/test_tuning_smoke.py, IMURUN_TUNING_SMOKE=1):

The 44 tokens/s that run reports is dominated by per-step overhead on 75-token samples (about 1.4 s per sample-step regardless of length at this size), so it is not a per-token speed. Extrapolating that per-step time (not measured): a 200-sample, 2-epoch run of the 1B is ~400 sample-steps ≈ 10–20 minutes of training here, plus ~1.5 minutes of export. The first real run records its measured throughput and later estimates use it.

CPU 4-bit QLoRA (the only path an 8B has here), same 1B smoke run with IMURUN_TUNING_SMOKE_METHOD=qlora-cpu: the baseline held-out eval of 4 samples took 23 s; the first optimizer step (one ~75-token sample) took 612 s, the second 376 s — about 0.2 tokens/s for a 1B. The run was then stopped (it was not going to finish usefully). That measured rate (0.25 tok/s × B) replaced the earlier 4.0 guess in feasibility.py; it is why 8B on this laptop is refused for any real dataset. Plain fp32 LoRA on the CPU (lora-cpu, which a 1B–3B could use without a GPU) was not measured; its speed in the planner is still a first guess.

The channel#

  chat turn ──► transcript (~/.imurun/sessions/<s>.jsonl)
                    │
  👍 / 👎 / Teach ──► POST /api/tuning/feedback ──► <sessions>/feedback/<s>.jsonl   (consent, per turn)
                    │
  files ──► ~/.imurun/tuning/datasets/*.jsonl|csv|md
                    │
                    ▼
     data.py: pair prompt+answer per consented turn, correction replaces answer,
              filters (tool_call, live_data, refusal, disclaimer, too_short, pii,
              duplicate, already_trained) ──► render with the engine's chat template
              (fish_chat_prompt) ──► train.jsonl / eval.jsonl (10 % by hash)
                    │
  scheduler.py (one job at a time; auto: off | nightly | when-idle | every-N)
     queued ─► preparing ─► training ─► evaluating ─► exporting ─► done
                 │  feasibility gate (ok / slow+allow_slow / impossible)
                 │  base weights download (size shown, confirm)
                 │           │ train.py in ~/.imurun/tuning/venv (below-normal priority)
                 │           │ pauses while any chat turn runs
                 │           │            │ held-out loss vs base + sanity prompts
                 │           │            │ regression ─► rejected (nothing exported, data not consumed)
                 │           │            │             │ merge LoRA ─► convert_hf_to_gguf ─► llama-quantize
                 │           │            │             │ + convert_lora_to_gguf (adapter, for engine E1)
                 ▼           ▼            ▼             ▼
            events: tuning.job {state, progress %, eta_s, loss, eval_loss, paused}
                                                          │
                                       done ─► manifest records the sample hashes
                                               registry.json + model catalogue (local model)
                                               tuning.ready {model_id, eval, action: "Use it"}
                                                          │
                                      user clicks ─► POST /api/models/select {id}   (never automatic)

Consent and privacy#

• Nothing is used unless you marked that turn. Thumbs-up = "learn this answer"; Teach = "the right answer is this" (your text replaces the model's). Thumbs-down alone is recorded but never trained on. rating: "clear" withdraws consent; the latest mark wins.
• Feedback lives next to the session it belongs to: ~/.imurun/sessions/feedback/<session>.jsonl (private file permissions).
• Nothing leaves the machine except downloads: the training stack from PyPI and the PyTorch index, the original model weights from Hugging Face, and two archives from the llama.cpp GitHub release (below). Training sets HF_HUB_OFFLINE=1 and drops HF_TOKEN from its environment. No telemetry, no uploads.
• PII filter: turns containing e-mail addresses, phone numbers, card numbers (Luhn-checked), API keys/tokens (OpenAI, GitHub, Slack, AWS, Google, HF, JWTs, password= style secrets) are dropped unless include_pii is on.
• The Hugging Face token (only needed for licence-gated repos) is write-only: GET returns "••••••••", never the value, and it is never logged.

Data filters#

Skill prefixes (/terse …) are stripped from prompts. Samples are single turns (user → assistant), without the app's system prompt.

Chat template. When the engine has that model (or a tuned descendant) loaded, prompts are rendered by the engine itself (fish_chat_prompt, read-only, under _gen_lock), so training sees exactly the text inference sees. Otherwise the trainer uses the Hugging Face tokenizer's apply_chat_template. Each row records rendered_by: fish | hf.

User files in ~/.imurun/tuning/datasets/: .jsonl / .json ({"messages": [...]} ending with an assistant message, or prompt+completion, question+answer, instruction+input+output), .csv with the same column names, .md with Q: / A: pairs or ### User / ### Assistant blocks. The same filters apply.

Training recipe#

LoRA r = 16, alpha = 32, dropout 0.05 on q,k,v,o,gate,up,down (Phi-3: qkv,o,gate_up,down); QLoRA NF4 + double quant when bitsandbytes works (else a bf16/fp16 base); gradient checkpointing; batch 1 × grad-accum 4; AdamW (paged 8-bit for QLoRA) with a cosine schedule, LR 2e-4, 5 % warm-up; 1–3 epochs (epochs, default 2); max sequence 512 tokens; eval on the held-out split twice per epoch with early stopping (patience 2) and the best checkpoint restored; checkpoints kept (last 2), so {"resume": job_id} continues an interrupted run. A new run continues from the last accepted adapter of the same base model, so learning accumulates.

Logs: ~/.imurun/tuning/runs/<run_id>/ → config.json, metrics.jsonl (every step: loss, learning rate, grad norm; every eval: eval_loss), result.json, train.log, export.log, job.json, control.json.

Acceptance. Before training: held-out loss and the sanity set on the untuned base (adapter disabled). After: the same. Rejected when the required prompt ("The capital of France is" → Paris) fails, when any sanity prompt the base answered now fails, or when held-out loss is more than 2 % worse than the base's.

Pausing. The trainer polls control.json after every optimizer step. The scheduler sets pause while any chat turn is running and clears it when the turn ends; a pause frees the CUDA cache but keeps the weights resident (the model is not unloaded). max_hours sets a deadline after which training stops early but still evaluates and saves. Export runs at below-normal priority and does not pause.

Export, and exactly what gets downloaded#

1. Training stack (python -m services.tuning.setup, once): a venv at ~/.imurun/tuning/venv with torch (CUDA 12.8 build from download.pytorch.org/whl/cu128 when an NVIDIA GPU is present, else the CPU build), transformers, peft, accelerate, safetensors, numpy, huggingface_hub, sentencepiece, protobuf (requirements-tuning.txt), and optionally bitsandbytes. Measured here: 2.8 GB torch wheel, 4.89 GB installed, 539 s. The app venv is never touched.
2. llama.cpp tools (with setup, or at first export), from the newest llama.cpp build release that ships a CPU binary for this OS (pinned after the first fetch in tools/llama.cpp/VERSION.json; override with IMURUN_LLAMACPP_TAG): the source archive github.com/ggml-org/llama.cpp/archive/refs/tags/<tag>.zip (41 MB; only the top-level *.py converters, the conversion/ package and gguf-py/ are extracted — no C++ is built) and llama-<tag>-bin-win-cpu-x64.zip (25 MB; only llama-quantize.exe and its DLLs are extracted). Measured tag: b12032. The pip gguf package is not used because it does not ship convert_hf_to_gguf.py and must match the converter's own gguf-py.
3. Original weights per model (on the first run for that model, size shown first, confirm required): *.safetensors, config.json, tokenizer files, from the official repo (catalogue.py), or — when the official repo is licence-gated and no token is set — the listed mirror that re-hosts the same files (e.g. unsloth/Llama-3.2-1B-Instruct; the original licence still applies). Measured: Llama 3.2 1B = 2.49 GB in 216 s from the mirror. Meta's duplicate original/ folder and Mistral's consolidated.safetensors are skipped.

Then per accepted run:

• export/adapter-<run>-f16.gguf — the LoRA adapter as GGUF (for engine request E1, REQUESTS.md; the engine cannot apply it yet).
• the adapter is merged into the base one safetensors shard at a time (W += alpha/r · B·A in fp32, cast back), so an 8B (5 GB shards) merges in ~10 GB of RAM; convert_hf_to_gguf.py writes an f16 GGUF (or Q8_0 / F16 directly), llama-quantize makes the chosen quant;
• result: models/<base-file-stem>-tuned-<run>-<quant>.gguf, e.g. Llama-3.2-1B-Instruct-tuned-r202611012300abc-Q4_K_M.gguf, registered as a local model. The merged safetensors and the f16 intermediate are deleted.

Settings (tuning section; GET/POST /api/tuning/settings)#

hf_token (stored in the huggingface section, write-only) is accepted by the same POST. Every automatic mode also requires: no running turn, at least two idle minutes, no other job, and a feasibility verdict of ok (or slow with allow_8b_slow).

API (all behind the bearer token)#

Events on /api/events:

CLI (works without the server): python -m services.tuning status, plan --model ID, build --model ID (dataset dry run), run --model ID [--confirm] [--allow-slow] [--max-steps N] [--min-samples N], and python -m services.tuning.setup [--probe].

Using the result#

When a run finishes you get tuning.ready. The tuned model is now in the Models list (GET /api/models, source: "local", tag tuned). Click Use it (or POST /api/models/select {"id": "<model_id>"}) to switch; switching back to the original is the same call with the original id. The channel never switches by itself. Delete a tuned model by deleting its GGUF in models/.

Files on disk#

~/.imurun/tuning/: venv/ (~4.9 GB), base/<model>/ (original weights), datasets/ (your files), built/ (assembled datasets), runs/<run>/ (job, config, metrics, checkpoints, adapter, export), tools/llama.cpp/, manifest.json, registry.json, hardware.json, setup.json, setup.log. Remove the folder to remove everything the channel created except the tuned GGUFs in models/.

Not real yet (do not claim)#

• The engine cannot load a LoRA adapter (request E1); each accepted run produces a full re-quantised GGUF.
• No UI yet: thumbs/teach buttons, the Tuning panel and the "Use it" card are frontend requests (REQUESTS.md). Today the channel is driven through the API or the CLI.
• Training pauses but does not release GPU memory held by the model weights; if the chat engine needs that VRAM during a pause it runs with what is left.
• 8B-class models cannot be tuned on this 4 GB laptop in practice (measured CPU 4-bit speed ~0.2 tok/s for a 1B; a normal dataset is refused).
• Multi-turn samples and the app's system prompt are not part of training samples yet.