9. Kernel tuning — [tune] and the per-app tune sidecar

llvm/daslib/llvm_tune turns one reference function into a tuned kernel family: a grid of code-generation permutations, a per-app record of which one wins, and a small policy rail that keeps an application honest about whether it is running tuned code. It sits on top of [llvm_code] (the JIT-time external code generator), so the family only generates under the LLVM JIT; on every other tier the reference body runs verbatim.

The design goal is that a shipped application reaches its own box’s floor with defaults that are data — a small JSON sidecar — rather than a fork of the kernels per machine. The winners are compile-time stamps: the front end reads the sidecar and stamps the winning permutation onto the function before codegen.

The sidecar is per-app: <app>.tune.json beside the app — the root script this process runs (the first .das on the command line), or the binary itself when there is none (standalone exe, embedded host). DAS_TUNE_MANIFEST overrides the location outright. A sidecar older than the running binary is stale and reads as absent everywhere — a rebuilt binary invalidates every measured winner, so copied-around stale files can never resurrect dead measurements.

Note

This is the [tune] framework (the [llvm_code] generator grid). An application’s own loop-hint profile (e.g. dasLLAMA’s [tuned] macro) tunes a different thing, but shares the same sidecar file — perm winners under "kernels", library runtime knobs under "runtime".

9.1. Overview

options gen2
require llvm/daslib/llvm_tune

// one reference function, a grid of [llvm_code] generator permutations,
// and a per-ISA fallback for an app with no sidecar yet.
[tune_perm(kstep = 1), tune_perm(kstep = 2), tune_perm(kstep = 4),
 tune(gen = "mylib::gemm_gen", fallback = "kstep2")]
def gemm(a, b, c : float?; n : int) : void {
    // reference implementation — runs on the interpreter, AOT, and any
    // target the generator declines. Never edited per box.
}

At compile time exactly one winner is stamped onto gemm:

  • the sidecar entry for gemm when one exists (this app has been tuned on this box, and the sidecar is not stale),

  • else the fallback= permutation (a generic per-ISA default),

  • else the reference body.

A harness discovers the winner by benching the whole grid, then records it in the sidecar. The next compile picks it up. Nothing about gemm’s callers changes — the annotated function is the real symbol.

9.2. The annotations

9.2.1. [tune_perm(...)]

One grid row. Its arguments are passed verbatim to the [llvm_code] generator (they become the generator’s parameters and fold into the JIT DLL cache key). suffix="..." overrides the auto-derived variant name; requires="feat" is a fallback-eligibility hint (a comma-separated AND of |-separated OR alternatives, matched against the host CPU features plus the DAS_JIT_*_FORCE_FEATURES overrides). tune_requires_ok(expr) evaluates such an expression on its own, and tune_pick_fallback(chain) runs the same rule over a ;-chain of suffix / suffix:requires entries — that is how a library whose permutations are generated, with no row to hang requires= on, still gets a per-ISA default.

9.2.2. [tune(gen="key", fallback="suffix")]

Closes a bracket of [tune_perm] rows (annotations apply in declaration order; tune consumes the rows banked before it). gen= is the [llvm_code] generator key. fallback= is a ;-separated chain tried in declaration order — the first permutation whose requires= hint passes on this box is the manifest-less default; reference forces the original body. The tuned function needs an explicit return type.

9.2.3. [tune_companion(fn="sibling", gen="key")]

Listed between the [tune_perm] rows and tune(...). Stamps a sibling function with the same permutation from the same manifest entry — the two-function stamp, so a kernel and (say) its repack-layout query can never desync, JIT-time declines included. The sibling is a plain function declared earlier in the same module, with an explicit return type.

9.3. The mode contract

The DAS_TUNE_MODE environment variable selects the compile-time behavior:

normal (default) Stamp exactly one winner onto the function (manifest > fallback= > reference). No variant stubs are emitted; a reference-row-only <name>_variants() registry still exists so harness code compiles in every mode.

tune / test Stamp the full grid as <name>__<suffix> clones plus the <name>_variants() registry (name → function-pointer rows). A harness benches them and records the winner (tune), or bit-exact-gates every variant against the reference row (test).

Note

A sidecar with no entry for a function falls through to its fallback= — a sidecar written before a kernel family landed must not silently drop that family to the reference tier. An explicit "reference" entry forces the original body.

9.4. Tuner wiring — [tune_scope]

A library that owns tuned kernels declares one scope:

[tune_scope(name = "mylib", tuner = "../harness/tune_mylib.das")]
struct private MyLibTuneScope {}

tuner= (resolved against the declaring file) names the harness that regenerates this library’s winners; the policy rail and --tune run it with DAS_TUNE_MANIFEST pointed at the app sidecar. covers= (optional, ;-joined module names) extends the scope’s demand beyond its declaring module: the completeness check requires a sidecar entry for every [tune] function AND every non-perm=-pinned [tuned] loop-hint kernel across the declaring + covered modules (dasLLAMA covers its math/quant modules, so first-start auto-tune sweeps loop hints too). The demand is AST-derived; a tuner that misses a demanded kernel re-tunes every start, and the startup warning names the missing kernels. version_of= (optional, "module/CONST") pins the scope to a library version: the sidecar’s provenance must record that int constant’s current value (under the lowercased constant name; version_key= overrides), so bumping the constant on kernel work invalidates every box’s winners. The annotation names that module by string only, so the declaring module must require it as well (suppress the unused-require lint), and the scope’s tuner must stamp the value with tune_provenance_note — an unstamped pin re-tunes on every start. The pin rides the same completeness check everywhere it runs — the policy rail, daspkg release --quick’s inherit gate, resolver adoption. The winners themselves live in the ONE per-app file — every library’s tuner upserts its own keys and preserves everyone else’s (that upsert is the isolation contract; “is this scope tuned” is per-key completeness, not file existence). Reading winners needs no scope at all — every [tune] resolves against the app sidecar.

Warning

The default-auto policy fires only for app roots that see llvm_tune — a library owning scopes must re-export it (require llvm/daslib/llvm_tune public), or its apps silently get no default policy. Re-export llvm_tune alone, not your module’s whole public surface (a blanket public on a module that also re-exports jobque_boost floods requirers with name ambiguities).

9.5. Application policy — [tune_policy] and --tune

Untuned does not start. Any application whose program root has a main and whose libraries declare tune scopes gets the auto policy by default — no annotation needed: an incomplete or stale sidecar tunes at startup and re-execs, so one launch already serves tuned kernels. The [tune_policy] annotation on main overrides the flavor (the auto / restart flavors prepend a runtime guard to its body):

[export, tune_policy(missing = "error")]   // dev mode: fail the compile instead
def main {
    // ...
}

missing

behavior when a scope’s sidecar entries are absent or stale

fallback

stamp fallback= silently (also what DAS_TUNE_POLICY=fallback — the CI kill switch — forces everywhere)

warn

loud compile-time banner with the exact tuner command

error

fail the compile with the same message — the dev mode

auto

the default, declared or not: tune at startup (a guard runs each incomplete scope’s tuner), then re-exec the process; the fresh compile stamps the winners. One launch, then it runs.

restart

like auto but no re-exec — tune at startup, then exit asking for a manual restart.

Programs whose root has no main never get the default — dastest-driven test files run [test] functions, so the test suite never tunes-on-start.

--tune after -- on the application’s command line forces the tune path even when the sidecar is complete (a re-tune; the flag is stripped from the re-exec so the child converges). DAS_TUNE_POLICY overrides the declared value — DAS_TUNE_POLICY=fallback is the CI kill switch.

Two further escapes exist for a run that must not mint. --jit-opt-level=0 flips the injected default to fallback, because winners raced under O3 codegen mean nothing at O0; a declared [tune_policy] is left alone, and DAS_TUNE_POLICY still overrides either way. tune_suppress_mint("KNOB") is the runtime twin, called from a library [init]: it suppresses the auto / restart mint process-wide, so untuned scopes keep their fallback stamps and the process runs. The name passed is the override the caller acts for, and it is printed verbatim in the suppression line. An explicit --tune outranks both.

Note

Tuning cannot happen mid-compile: the tuner is a separate daslang process, and adopting its winners means new compile-time stamps regardless. That is why auto tunes at runtime and re-execs into a fresh compile rather than stamping in place — which also keeps the winners cross-module-safe (a required library’s kernels are stamped at their own [tune] time, not mutated after the fact). Re-exec is why an auto application’s main must return void or int.

9.6. Standalone builds

A frozen artifact must not demand or run tuning:

  • Cross-box artifacts (policies.tune_frozen — set by the -aot C++ generator and by dastest’s AST serializer) are fully tune-free — every tune annotation is inert, only the reference-row registries are emitted. Per-box stamps would otherwise desync the artifact against the box that consumes it.

  • AOT-consuming runs (policies.aot) are tune-free only when the JIT is off: a stamp changes the function’s semantic hash, so a stamped function would fail the AOT link. Under -jit the JIT supersedes AOT bodies and tuned stamps stay live.

  • Standalone exe (llvm-jit -exe) still stamps — an exe built beside a sidecar ships those winners (a local-use artifact by definition), and one built without ships the generic fallback= stamps — but the policy rail is dead (no [tune_policy], no --tune). The exe DOES get the status [init], so the artifact self-reports its baked stamps (tune_status() / log_tune_status work inside a standalone exe). A native exe carrying any [llvm_code] kernel also targets the build box (CPU and feature gates, decided before the target-flag pass) so the stamped generators actually emit instead of declining to their reference bodies on the generic-CPU rail; a kernel-free exe stays generic/redistributable.

One more compile is inert for a different reason: under the host’s -documentation policy — a documentation or reflection root — every tune annotation and the policy rail are off. Ask with is_building_documentation().

daspkg release applies “untuned does not start” to artifacts at build time: the -exe build’s release-deps JSON reports every scope with per-key completeness (tune_scopes_status), and every scope that declares a tuner re-mints on every release — a frozen exe never tunes, so an unmeasured winner would ship forever. --quick is the only mode that inherits an existing sidecar, and only a complete, fresh one; an incomplete or stale sidecar mints even under --quick. A tuner that refuses (the noise gate, a failed validation) fails the release outright rather than shipping fallbacks quietly. daspkg then rebuilds so the exe bakes the measured winners, and ships the sidecar beside the exe as <bundle>.tune.json (touched newer than the exe — the "runtime" knob section travels with the artifact, and the file documents the baked winners). The sidecar it measures against is the source-side one beside the app script; the bundle gets a byte-identical copy.

9.7. Sidecar location and staleness

The sidecar is <app>.tune.json beside the app: utils/myapp/main.das reads and writes utils/myapp/main.tune.json; a standalone exe (or an embedded host with no .das on its command line) uses the binary’s own stem. The app identity is the first .das argument before -- on the process command line — macro time and runtime read the same argv, so compile-time stamps and the harness write API always agree on the file. No declaration is needed; DAS_TUNE_MANIFEST overrides the location, and set_tune_manifest_runtime_path points just the get/set APIs elsewhere for self-managed harnesses.

A sidecar whose mtime predates the running binary’s is stale: it reads as absent (stamps fall back, the policy rail re-tunes), and the first tune_manifest_set resets it to a fresh document. Measurements never outlive the binary that made them. A sidecar carrying another box’s identity is stale too — measurements are a property of the box — unless provenance.applied_box names this box, which is how a scope resolver (below) adopts a compatible sibling box’s mint deliberately. That identity is the five-field platform|arch|model_id|os_build|cpu string tune_box_identity() builds, and the compare runs through box_match_key, which folds the volatile Windows os_build UBR, so a monthly cumulative update does not re-tune while a real OS-version change does. A scope declaring version_of= adds a third axis: a sidecar minted at another library version reads incomplete for that scope alone.

Every “untuned” refusal names its reason via tune_sidecar_verdict (TuneSidecarReason) — absent (no sidecar at the path), stale_binary (both dates), foreign_box (both identities), unreadable (not a tune sidecar), version (both values), missing (the kernel names), so a bare “untuned” never leaves the operator to diff provenance by hand. And when DAS_TUNE_MANIFEST points at a file that reads untuned, the compile prints one loud warning per scope instead of silently stamping fallbacks — an explicit manifest disables the policy rail, so nothing else would say so, and a measurement run believing its winners are live while stamping fallbacks is the exact hazard the sidecar model exists to prevent.

Two seams let a supervisor or a network service participate:

  • tune_set_scope_resolver(fn) — registered from an [init] (which must run before the guard at the top of main), consulted by the auto/restart policy guards before spawning a scope’s tuner. A resolver that can satisfy the scope another way (dasLLAMA’s exchange client downloads a matching per-box sidecar from dasllama.io) returns true; completeness is re-checked, never trusted, and --tune never consults it.

  • tune_interrupt_requested() — true while the file named by DAS_TUNE_CONTROL exists. Tuners poll it between kernel families and abort without minting; the measurement in flight always completes. (A tuner that spans multiple processes end-gates each process’s own write, so an interrupt in a later process leaves earlier processes’ mints on disk.)

9.8. Runtime status — tune_status()

tune_status() returns one row per [tune] function — its winning suffix, the source (manifest / fallback / reference), the scope, and the sidecar path. It is populated when the application declares [tune_policy]. log_tune_status("myapp") is the ready-made “am I tuned?” surface — it logs the table at LOG_INFO (<n>/<total> kernels tuned for this box, one line per function, plus a --tune hint when any kernel is on a non-manifest tier), and is a no-op when the table is empty. Call it at startup:

log_tune_status("myapp")   // or iterate tune_status() yourself

9.9. What a tune shows

A tune is minutes of work spread over a chain of child processes, so before the wait the framework says both that it is tuning and that the process will restart afterwards. During the wait it shows a progress display:

[#####-------------]  9/25   dot_q8q8   finalists 47/80   4 live   1m12s

Outer counter (kernels done), the kernel being tuned, the screening pass and its round counter, how many grid rows are still in contention, and observed elapsed. Nothing there is predicted — the segments are far too uneven for an ETA to mean anything, so none is offered.

Three verbosity levels, selected by --tune-quiet / --tune-verbose on the application (or DAS_TUNE_VERBOSITY directly):

silent Nothing at all.

normal (default) The display, and the closing summary.

verbose Every line the tuner prints, and no display.

9.9.1. Progress events

The display is driven by events the tuner emits as prefixed lines on stdout — a pipe is the only channel that spans the process chain:

@tune plan scope=dasllama_kernels total=25
@tune begin name=dot total=20
@tune step i=5 phase=narrowing live=12
@tune end name=dot winner=vec8_u2 verdict=holds

A process renders the display when its own stdout is a terminal, and otherwise forwards the events untouched, so whatever is capturing it (a supervisor, a parent tuner) can render instead. Forwarding is unconditional; verbosity gates only what a human sees. That is what lets silent mean literally nothing on a terminal while a supervised run still records the full event stream.

Raw tuner output is muted only while a display is actually live, so a tuner that emits no events keeps printing exactly as it always did.

A harness emits events with tune_progress_plan / _kernel_begin / _kernel_step / _kernel_end. A different front end (a GUI, a status page) consumes them with tune_progress_feed and renders tune_progress_line.

9.10. The noise gate

A winner raced on a non-silent box is a random number, and the sidecar is a file — one noisy mint persists and reads as a code regression later. A harness therefore probes the box before, during, and after its races: it times one fixed kernel for a dozen rounds at the real round shape, computes the coefficient of variation, and refuses to tune when the cv exceeds the threshold — nonzero exit, nothing written, the measured cv printed so you can watch it fall as you quiet the box. A failed probe gets one settle-and-retry first (a transient the run itself caused fades in seconds; outside noise does not), and a failed end probe discards the whole run: winners only reach the sidecar through a gate that just passed.

The framework carries the policy and the math; the harness carries the probe:

  • tune_noise_threshold_pct(paranoid) — the cv ceiling: 2% normal, 1% paranoid; DAS_TUNE_NOISE_CV=<pct> overrides for calibration.

  • tune_noise_override() — DAS_TUNE_NOISE_OVERRIDE=1 turns refusals into passes at BOTH gates: a failed noise probe (stamped noise: overridden) and a failed validation verdict (stamped validation: overridden); the escape always leaves a mark. It is one flag by design — “mint no matter what” — so quieting a noise refusal also waives reproduction.

  • tune_median(samples) / tune_cv_pct(samples) — rank finalists by the median of their finalist rounds (print best-of alongside); a best-of far under its own median is a per-kernel noise flag.

  • tune_provenance_note(key, value) — attach the noise verdict (and any other measurement condition) to the "provenance" section of every subsequent sidecar save in the process.

Two placement rules from the dasLLAMA harness (the worked consumer): a winner must beat the shipped fallback by more than the measured noise floor or the fallback keeps the seat — deterministic beats lottery; and the closing probe runs before any GPU race in the same process, so the bracket covers exactly the window that produced the winners. (GPU races block the thread, and a thread that blocked wakes cold — the same E-core transient a sleep causes; it recovers under a couple of seconds of sustained load, but a post-race gate would pay that settle on every mint for a window whose noise cannot touch the CPU medians anyway.)

After the sweep, the harness validates itself: a heavy subset re-races twice at the same budget and the verdict is same-window ORDER (TuneValidateRow + tune_validate_verdict) — a winner fails only when a re-race rejects it outright, or the SAME challenger beats it past the reproduce band in BOTH windows. Anything else — an inside-floor flip, a one-window flip, differing challengers — is reproduction (ties break deterministically: baseline first, then grid order). Median drift between race and re-race is stamped (validation_max_drift_pct) and noted, never failed: a box’s level moves with load history while same-window order holds, so cross-window medians prove nothing. A failed verdict fails the mint as a whole unless the override above mints through it, stamped. Two budgets exist: default and paranoid (3× the rounds, a 1% cv ceiling, a tighter drift note). There is deliberately no fast race mode: a short race cannot resolve sub-2% twins, so its winners are lottery tickets; the debugging concession is accepting an existing sidecar, never racing cheaply.

9.11. Adding a kernel family

  1. Write the reference def with an explicit return type, and register its generator key under [llvm_code] (llvm/daslib/llvm_jit_code.das).

  2. Add the [tune_perm] grid, then tune(gen=, fallback=); any [tune_companion] rows go between the two.

  3. A library that already declares a [tune_scope] absorbs the new family automatically — same sidecar, same tuner. Otherwise declare one scope on a dummy struct in that module.

  4. Teach the harness to bench <name>_variants() and record the winner with tune_manifest_set("<name>", winner). A demanded kernel with no sidecar entry re-tunes the whole scope on every start, so a family the harness does not sweep yet still needs its shipped fallback recorded.

9.12. Writing a harness

A tuner is an ordinary [export] def main compiled with DAS_TUNE_MODE=tune, so the <name>_variants() registry holds the full grid as function pointers. It benches them (see the measurement discipline in modules/dasLLAMA/tune_for_this_box.md — interleaved A/B, correctness-gate every candidate, best-of-N, confirm the winner), then records the winner:

let ok = tune_manifest_set("gemm", winning_suffix)

tune_manifest_set writes to the DAS_TUNE_MANIFEST the policy rail sets when it spawns the harness, so the winner lands in the app’s sidecar. It UPSERTS: other functions’ entries and other sections survive. A following normal compile stamps it.

9.13. Sidecar format

A sectioned JSON object — perm winners under "kernels", library runtime knobs under "runtime" (owned by the library that reads them), the race tables behind each multi-variant winner under "race", and "provenance" (who wrote it) refreshed on every write:

{
    "kernels" : { "gemm" : "kstep4", "gemm_gemv" : "reference" },
    "provenance" : { "binary" : "...", "platform" : "windows", "arch" : "x86_64",
                     "noise" : "ok", "noise_probes" : "start cv 0.40%; mid1 cv 0.14%; end cv 0.28%" }
}

It is a per-app, per-box artifact — gitignored (*.tune.json), and any change re-keys the JIT DLL cache automatically (the winning permutation’s args fold into the DLL basename).

Every mint is also archived to <home>/.tune-history/<box>/ (or DAS_TUNE_HISTORY): successful mints copy the whole sidecar — provenance and race tables ride inside, so each archive is self-contained — and failed mints leave a marked .FAILED.json stub. Nothing in the history is ever deleted; it is the box’s longitudinal health record, and tune_history_archive is the framework call a mint wrapper makes. A re-mint also snapshots the previous sidecar to <sidecar>.bak and prints the diff — winner changes with their margins from the new race tables, plus the median time shift over shared rows. Uniform shift = box state; scattered past-floor flips = a noisy mint; same-direction twin flips = an estimator change.

See also

modules/dasLLAMA/tune_for_this_box.md — a worked application of this framework (the dasLLAMA gen GEMM family), including the measurement discipline that separates a real win from a benchmark artifact, and the dasllama-server / ask / wav2txt trio sharing one box’s winners.