17.1. dasLLAMA LLM inference: models, sessions, generation, chat

CPU large-language-model inference in pure daslang: load a GGUF model, tokenize, run the transformer, sample — or hold a full chat — validated token-for-token against llama.cpp on every supported family. Run with -jit; examples/dasLLAMA/run.das and chat.das show the canonical program shape.

Supported model families (GGUF — fp32 / f16 / q8_0 / q4_0 / mxfp4 weights read directly; K-quant files such as Q4_K_M / Q5_K_M / Q6_K run on native K-quant kernels):

  • Llama — Llama-2 / TinyLlama, Llama-3.1 / 3.2, Mistral-7B-Instruct, SmolLM2, plus llama2.c .bin checkpoints

  • Qwen — Qwen2.5, Qwen3 (QK-norm), Qwen3.5 / Qwen3.6 (hybrid Gated-DeltaNet attention, incl. the 35B-A3B MoE); MoE: Qwen1.5-MoE (routed + sigmoid-gated shared expert), Qwen3-30B-A3B (routed-only, renormalized top-k); vision + omni: Qwen3-Omni-30B and dense Qwen3-VL (M-RoPE image positions, deepstack), Qwen2.5-Omni / VL (window-attention ViT) — images and audio in chat

  • Phi — Phi-3.5-mini

  • Gemma — Gemma-2, Gemma-3 (per-layer sliding-window patterns, vision via the SigLIP mmproj), Gemma-4 (12B / 31B dense, the 26B-A4B MoE, and the E2B / E4B edge series with per-layer embeddings + cross-layer KV sharing; vision and audio in chat via the family mmprojs)

  • gpt-oss — gpt-oss-20b (attention sinks, native MXFP4 experts, YaRN long context, Harmony chat format)

The architecture is picked from GGUF metadata at load — the same program runs any of these.

Hands-on tutorials (overview): the problem statement, hello, generation, chat and templates, sampling, sessions and memory, performance, the architecture registry.

17.1.1. Type aliases

DlimImageInfo = DlimImageInfo

typedef DlimImageInfo = dasllama_image::DlimImageInfo aka DlimImageInfo

GpuTierStatus = GpuTierStatus

typedef GpuTierStatus = dasllama_gpu_tier::GpuTierStatus aka GpuTierStatus

GpuTierWant = GpuTierWant

typedef GpuTierWant = dasllama_gpu_tier::GpuTierWant aka GpuTierWant

GpuModelMarks = GpuModelMarks

typedef GpuModelMarks = dasllama_gpu_resident::GpuModelMarks aka GpuModelMarks

17.1.2. Types

The engine types the API below works with. They are created and consumed by the functions of this module; their remaining fields are engine implementation detail.

Model

A loaded model: weights, config, tokenizer, and the architecture’s blocks and chat template, as produced by load_model. User code touches config (e.g. cap config.seq_len before create_session on large-context models) and arch (the GGUF architecture name).

Session

One generation stream over a model: the KV cache, scratch buffers, sampling RNG, and the current position n_past. logits holds the distribution produced by the last eval. A model serves many independent sessions.

BatchWorkspace

Caller-owned scratch for eval_batch: the batched activation buffers a step of B sessions shares. One per concurrent batch, reused across calls; holds no session state — positions, caches and logits stay in the sessions.

KVPool

A caller-owned paged KV-cache pool (create_kv_pool): sessions created over it allocate fixed-size page groups on demand, so cache memory tracks the actual context instead of the full seq_len slab. One pool serves many sessions; an eval_batch batch must share one pool. Keep it alive and in place while its sessions live.

PrefixCache

A page-granular prefix cache over one pool’s sessions (create_prefix_cache): finished streams donate their KV pages keyed by a chained page hash of the token history, and later requests attach the longest cached prefix instead of re-prefilling it. Pages are refcounted with the pool; an LRU budget bounds retention.

KVDtype

Per-session KV-cache codec picked at create_session/create_chat: f16 — the default, half the KV bytes and faster deep-context decode (stores clamp to ±65504); f32 — the bit-exact reference; q8_0 — block-quantized (llama.cpp -ctk/-ctv q8_0), half the f16 bytes again and near-lossless in practice; needs head_size and kv_dim to be multiples of 32 (checked at create); tq4 — the FWHT-rotated ternary codec, smaller still.

QuantMode

Weight representation picked at load: fp32 — the token-exact reference; q8 — int8 quantization, the fast CPU path (K-quant/mxfp4/Q4_0 files keep their native planes on the same rails); q4_0 — the legacy requant tier, 4-bit blocks, smallest footprint.

SamplingParams

Sampling knobs: temp (<= 0 selects greedy argmax), top_k (0 = no cutoff), and repetition penalty (1.0 = none) applied over the last penalty_last_n generated tokens. The defaults are greedy — SamplingParams() reproduces argmax exactly.

Stats

Timing of the last generate/respond call: n_prompt/n_gen token counts, ttft_s (seconds to first token), and prefill_tps/gen_tps throughput in tokens per second.

LlmCaps

What the model honestly supports at the chat layer, as returned by caps: system_prompt is false for architectures with no system role (gemma), where the chat layer silently folds the system prompt into the first user turn. Grows as gaps surface.

ChatSession

A conversation over a model: its session, the resolved chat template, and the running transcript in history. Create with create_chat, then drive with add_user + respond.

ThinkStream

The incremental reasoning/content splitter for a streamed reply (make_think_stream): feed decoded pieces through think_feed, flush with think_finish. Holds partial reasoning markers back across chunk boundaries; a non-thinking family yields a pass-through stream.

ThinkBudgetMarks

The token marks a reasoning budget acts on (think_budget_marks): open the token a reply writes to open its reasoning span (empty where the prompt opened it), close the tokens a scheduler forces to end the span at the request’s think_budget - a family’s close special, or harmony’s end mark and final-channel header - and opened whether the generation prompt left the reply inside the span. An empty close means no budget can act.

ThinkSplit

One reply split at its reasoning boundary, as returned by split_reasoning: reasoning holds the thinking span (empty when the model answered directly), content the answer.

NothinkStopGuard

The instruct-mode stop guard over a family’s channel markers (make_nothink_guard): a loop that samples tokens itself checks turn_stop_ids first, then asks nothink_stop_here - a marker before the reply’s first content piece is a leading thought the reply matcher splits, one after content ends the turn. Unarmed (every token passes) on a thinking turn and on families with no such markers.

ToolReply

A fully classified tool-capable reply, as returned by parse_calls: the reasoning span, the content, and the calls the model made (empty when it answered directly).

ToolCall

One parsed tool call: the function name, its args normalized to JSON object text, and the wire id where the family carries one (mistral; empty elsewhere).

AudioTower

A loaded audio encoder: Whisper-family encoder weights plus the model-specific projector tail, as produced by load_audio_tower from an mmproj GGUF. Pass it to create_chat to enable add_user_audio turns.

VisionImage

A decoded RGB8 image: width, height, and the pixel bytes in rgb. Produced by load_image_rgb (the optional dasllama_vision_io companion) or built directly; add_user_image / encode_image letterbox and normalize it to the model’s geometry.

VisionEmbedder

A loaded vision embedder of whatever family the mmproj GGUF turned out to be (gemma4uv — the gemma-4 dense embedder; gemma4v — the gemma-4 E-series ViT tower; gemma3v — the gemma-3 SigLIP tower; qwen3v — the Qwen3-Omni / dense Qwen3-VL ViT, deepstack included; qwen25v — the Qwen2.5-Omni/VL window-attention ViT), as produced by load_vision_embedder, which sniffs the family. Pass it to create_chat to enable add_user_image turns; vision_proj_dim must match the decoder’s embedding width (checked at create).

VisionState

Caller-owned scratch for the embedder forward of whichever family is carried: the buffers encode_image reuses across calls. One per embedder user; holds no image state between calls.

AudioEmbedder

A loaded audio encoder of whatever family the mmproj GGUF turned out to be (gemma4a — the gemma-4 E-series Conformer; or a whisper-class tower that load_audio_tower also serves — qwen2-audio, qwen2.5-omni, ultravox, voxtral), as produced by load_audio_embedder, which probes the file; audio_probe_proj_dim answers 0 where absence is an answer. audio_proj_dim must match the decoder’s embedding width. The server’s media worker owns one per armed slot.

AudioState

Caller-owned scratch for the audio encoder forward of whichever family is carried: the buffers encode_audio reuses across calls. One per embedder user; holds no clip state between calls.

DlimImageInfo

One prepared image beside a GGUF, as listed by dlim_inventory: file, path, bytes, the image version, its bake identity, and a verdict — CURRENT loads; STALE vN is an older image version; OTHER a different bake configuration or box; FOREIGN a different flavor.

GpuTierStatus

Snapshot of the GPU tier in force (gpu_tier_status): the device name, vram_bytes held, and the resident/streamed layer counts, dense planes and classifier residency the armed backend reports.

GpuTierWant

A programmatic GPU tier request (set_gpu_tier_want): auto_tier asks for the measured-best rail set; moe_layers/moe_stream/vram_mb and the per-rail flags override it. The core DASLLAMA_GPU_* env knobs override the matching fields when present. GpuTierWant() is a zero want — no device.

GpuModelMarks

One model’s complete per-model GPU tier state while it is NOT the installed model (a multi-model host keeps one per slot): its routing marks plus the resident driver’s activation. Mint with gpu_model_marks_init; gpu_slot_capture fills it after the model’s load, moe_gpu_model_marks_save/_restore swap it in and out on a switch.

17.1.3. Model loading and sessions

attach_mtp_drafter(model: Model; path: string ): bool

Attach the assistant drafter at path to model for speculative decode; false where the driver refused it (a drafter made for another target) or no driver carries one.

Arguments:

  • model : Model

  • path : string

caps(model: Model ): LlmCaps

What model honestly supports at the chat layer (see LlmCaps) — e.g. gemma has no system role, so the chat layer folds the system prompt into the first user turn; system_prompt is false there so callers can surface it instead of being silently absorbed.

Arguments:
create_batch_workspace(model: Model ): BatchWorkspace

Create the caller-owned scratch that eval_batch steps through — one per concurrent batch, reused across calls (buffers grow to the largest batch seen). Holds no session state: the sessions keep their own positions, caches and logits.

Arguments:
create_kv_pool(model: Model; page_rows: int64 = 64; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): KVPool

Create a caller-owned PAGED KV pool over model’s cache geometry. Sessions created over it allocate cache pages of page_rows positions on demand instead of the full seq_len slab up front, so many sessions share one elastic pool. Keep the pool alive as long as sessions live.

Arguments:

17.1.3.1. create_session

create_session(model: Model; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): Session

Create a fresh session (KV cache + scratch) sized to model.config.seq_len — one model, many independent conversations. kv_dtype picks the KV-cache codec: f16 (default, near-lossless) halves bytes, q8_0 halves again (needs head_size/kv_dim % 32 == 0).

Arguments:
create_session(model: Model; pool: KVPool ): Session
load_model(path: string; mode: QuantMode = dasllama_common::QuantMode.fp32 ): Model

Load a model AND its tokenizer from a GGUF file — architecture and tokenizer backend are auto-selected from metadata; mode picks the weight quantization. Q8 loads cache a PREPARED IMAGE beside the gguf for millisecond reloads (DASLLAMA_IMAGE=0 disables); under an active Metal mode the image is the BLOB-ONLY metal flavor (CPU inference against it panics).

Arguments:
mtp_capable(model: Model ): bool

Whether model can run a speculative round: a NextN head of its own, or an assistant drafter attached - what a scheduler’s mtp asks for.

Arguments:
mtp_drafter_sidecar(model_path: string ): string

The assistant drafter file a model ships beside it (gemma-4: mtp-<stem>-Q8_0.gguf), “” where there is none or no driver carries one. A model with no NextN head of its own speculates through it once attached.

Arguments:

  • model_path : string

release_kv_pages(session: Session )

Return session’s KV pages to its pool (no-op on flat sessions). The normal shape is release + delete; a released session stays alive but loses its cached context — to reuse it, also reset session.n_past to 0.

Arguments:
setup_dasllama_jobque()

Configure the job queue for dasLLAMA’s fork/join matmul dispatch: pooled fork contexts, batched dispatch, the worker spin-before-park window (jobque_spin_us; 0 disables). Call INSIDE with_job_que(), before the first generate/eval. The fork-context pool is PER CONTEXT: a new_thread body that runs inference calls this again, or every parallel kernel clones the whole program (90x slower).

with_dasllama_jobque(blk: block<():void> )

Run blk inside a fresh with_job_que() that setup_dasllama_jobque() has configured — the one-call form of the pair. The queue lives for the block only; a program that keeps its own queue calls setup_dasllama_jobque() inside it instead.

Arguments:

  • blk : block<void>

17.1.4. Prefix cache

create_prefix_cache(max_groups: int64 = 0; max_states: int64 = DEFAULT_PREFIX_STATES; max_state_bytes: int64 = 0 ): PrefixCache

A prefix cache for one create_kv_pool pool: streams donate KV pages (prefix_insert), later requests with the same prefix attach them (prefix_attach); max_groups caps the pages (0 = unbounded). A recurrent model keeps whole checkpoints instead: max_states within max_state_bytes (0 = the count alone).

Arguments:

  • max_groups : int64

  • max_states : int64

  • max_state_bytes : int64

prefix_attach(cache: PrefixCache; pool: KVPool; session: Session; prompt: array<int64> ): int64

Attach the longest cached prefix of prompt to a FRESH paged session of pool: matched pages join the session’s block table and n_past advances past them, so the caller prefills only the tail. Returns the matched count, capped one token short of the prompt.

Arguments:
prefix_chain_list(cache: PrefixCache ): array<PrefixChain>

Snapshot of the cache’s donated chains for dashboards: per donation — page-covered token count, live pages, hit count, born/last-hit ticks, and the caller-provided preview.

Arguments:
prefix_checkpoint_at(cache: PrefixCache; pool: KVPool; session: Session; prompt: array<int64>; matched: int64; stable_at: int64 = 0 ): int64

Where a recurrent session’s prefill of prompt should stop for a prefix_insert, after prefix_attach matched matched tokens: stable_at (render_turn_marked’s opening) when the caller knows it, else the longest opening an earlier checkpoint shares. 0 = nowhere - always on a model with no recurrent state.

Arguments:
prefix_held_groups(cache: PrefixCache ): int64

Pages the cache currently holds (== pool groups retained for reuse).

Arguments:
prefix_insert(cache: PrefixCache; pool: KVPool; session: Session; tokens: array<int64>; preview: string = ""; opening: bool = false )

Donate a session’s KV pages to the cache: tokens is its EVALED history (the first n_past rows exist), every full page not already cached survives release_kv_pages, preview labels the chain. A recurrent model donates one checkpoint at n_past: opening marks one taken between prefill calls, kept ahead of finished turns’ tails.

Arguments:
prefix_match_len(cache: PrefixCache; pool: KVPool; prompt: array<int64> ): int64

How many leading positions of prompt a prefix_attach would attach right now, attaching nothing: a probe for a caller deciding what a request must bring (a media span’s rows). 0 on a recurrent model’s cache.

Arguments:
prefix_release(cache: PrefixCache; pool: KVPool )

Release every cached page back to pool and clear the cache (pages still used by live sessions stay alive until those sessions release them). Call before deleting the pool.

Arguments:

17.1.5. Tokenizer

decode(model: Model; ids: array<int64> ): string

Decode a token-id sequence back to text with the model’s tokenizer.

Arguments:

  • model : Model

  • ids : array<int64>

encode(model: Model; text: string; add_special: bool = true; parse_special: bool = false ): array<int64>

Encode text to token ids with the model’s tokenizer. add_special prepends BOS where the model expects one. parse_special is reserved and not yet honored — special tokens reach the model as atomic ids from the chat layer’s template renderer, never by spelling them in text.

Arguments:

  • model : Model

  • text : string

  • add_special : bool

  • parse_special : bool

piece(model: Model; id: int64 ): string

Decode a single token to its text piece — the streaming counterpart of decode.

Arguments:

  • model : Model

  • id : int64

17.1.6. Evaluation and sampling

eval(model: Model; session: Session; tokens: array<int64> )

THE eval primitive: run tokens at the session’s current position and advance it. Prefill = eval(prompt); each generation step = eval([token]) — the same call at different batch sizes. Logits land in session.logits.

Arguments:
eval_batch(model: Model; ws: BatchWorkspace; sessions: array<Session?>; tokens: array<int64> )

One synchronous batched decode step: row i evals tokens[i] at sessions[i]’s current position, advancing each by one — B conversations through ONE pass of the weights (GEMVs batch into GEMMs). Sessions must be distinct, same-geometry, one pool if paged.

Arguments:
eval_embd(model: Model; session: Session; embd: array<float>; npos: int64; non_causal: bool = false )

eval’s embedding-input twin: prefill npos pre-built embedding rows (npos × dim, token-major) at the session’s current position and advance it — the multimodal splice entry. non_causal marks the whole call as one image span (gemma vision); audio rows stay causal.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • non_causal : bool

eval_embd_body(model: Model; session: Session; embd: array<float>; npos: int64; span_lo: int64; span_hi: int64; grid: int2 )

Prefill npos rows built by media_body_rows: [span_lo, span_hi) a non-causal span (empty = every row causal), roped on a merged image grid where one is given.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • span_lo : int64

  • span_hi : int64

  • grid : int2

eval_embd_span(model: Model; session: Session; embd: array<float>; npos: int64; span_lo: int64; span_hi: int64 )

eval_embd for a prompt carrying one NON-CAUSAL image span: rows [span_lo, span_hi) prefill with every query attending the whole span, the text around them causally — the gemma vision decode shape (non-causal media chunk).

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • span_lo : int64

  • span_hi : int64

eval_embd_span_mrope(model: Model; session: Session; embd: array<float>; npos: int64; span_lo: int64; span_hi: int64; grid: int2 )

eval_embd_span’s qwen mrope twin: the span rows rope as a grid-shaped merged image (position advance max(grid.x, grid.y), tracked on the session for every later eval); same span mask, only the angles differ. Panics without rope.dimension_sections.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • span_lo : int64

  • span_hi : int64

  • grid : int2

eval_segments(model: Model; sessions: array<Session?>; toks: array<array<int64>> )

Several sessions’ prompt chunks as ONE prefill: segment i runs toks[i] at sessions[i]’s position and advances it, each segment’s logits landing in its own session. Where the serving GPU prefills, the chunks share one window; elsewhere each runs as eval would. Distinct sessions, no empty chunk; no segments is a no-op.

Arguments:

  • model : Model

  • sessions : array< Session?>

  • toks : array<array<int64>>

media_body_rows(model: Model; session: Session; head: array<int64>; rows: array<float>; n_media: int64; src_w: int64; tail: array<int64>; embd: array<float> )

One prefill body’s rows for eval_embd_body on session: head text, n_media media rows of width src_w, tail text.

Arguments:

  • model : Model

  • session : Session

  • head : array<int64>

  • rows : array<float>

  • n_media : int64

  • src_w : int64

  • tail : array<int64>

  • embd : array<float>

sample(session: Session; params: SamplingParams ): int64

Sample the next token from session.logits per params: penalties, then temperature/top-k/top-p/min-p and a CDF draw — or greedy argmax when params.temp <= 0 (SamplingParams() defaults are greedy).

Arguments:
set_seed(session: Session; seed: int )

Seed the session’s sampling RNG for reproducible generation.

Arguments:
stats(session: Session ): Stats

Timing of the most recent generate/respond call on session: prompt/generated token counts, time to first token, prefill and generation tok/s.

Arguments:

17.1.7. Generation

generate(model: Model; session: Session; prompt: array<int64>; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

Stream-generate up to max_tokens from prompt, invoking the trailing block per token with (id, piece); return false from the block to stop early. Prefills the prompt in one eval, then samples one token at a time; returns the number of tokens emitted.

Arguments:

  • model : Model

  • session : Session

  • prompt : array<int64>

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

17.1.7.1. generate_embd

generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

generate’s embedding-prefill twin: prefill npos pre-built embedding rows (the multimodal splice — see eval_embd), then stream-sample exactly like generate. The chat layer’s audio turns run on this; use it directly for custom multimodal prompts.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; span_lo: int64; span_hi: int64; blk: block<(id:int64;piece:string):bool> ): int64
generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; span_lo: int64; span_hi: int64; grid: int2; blk: block<(id:int64;piece:string):bool> ): int64

17.1.8. Embeddings

embed(model: Model; text: string; pooling: int64 = -1; dims: int64 = 0 ): array<float>

L2-normalized sentence embedding of text (model.config.dim floats, or the leading dims renormalized - the Matryoshka read) from the decoder’s post-final-norm hidden rows, pooled the way the file’s pooling_type asks (mean; the last row for Qwen3-Embedding) unless pooling names one (POOLING_MEAN / POOLING_CLS / POOLING_LAST; -1 = the file’s).

Arguments:

  • model : Model

  • text : string

  • pooling : int64

  • dims : int64

17.1.9. Vision and audio encoders

encode_audio(embedder: AudioEmbedder; scratch: AudioState; samples: array<float>|array<float>#; out: array<float> ): int64

encode_image’s audio twin: 16 kHz mono samples through the carried family’s encoder into out = npos × proj_dim soft tokens (returns npos). A scheduler owning its own embedder calls this; the server’s media worker is the worked example.

Arguments:
encode_image(embedder: VisionEmbedder; scratch: VisionState; img: VisionImage; tag: string; out: array<float> ): int64

The whole image path in one call — geometry, letterbox, normalize, encode — into out = npos × proj_dim soft tokens (returns npos). add_user_image is this plus queueing onto a chat turn; a scheduler owning its own embedder calls this. tag names the vision dump.

Arguments:

17.1.10. Chat

add_assistant(model: Model; chat: ChatSession; text: string )

Inject a KNOWN assistant reply (no generation): prefill the pending user turn and text into the KV cache, then close the turn — like respond but with a supplied reply. The shape a stateless server needs to replay history. Precondition: a user message is pending.

Arguments:
add_user(chat: ChatSession; text: string )

Queue a user message for the next respond.

Arguments:
add_user_audio(chat: ChatSession; samples: array<float>|array<float># ): auto

Queue audio (16 kHz mono f32 PCM) for the next respond — encoded to soft tokens immediately and spliced at the head of the turn before any add_user text. Needs a chat created with create_chat(model, tower); call inside with_job_que().

Arguments:

  • chat : ChatSession

  • samples : option<array<float>| array<float>#>

add_user_audio_rows(model: Model; chat: ChatSession; rows: array<float>; n_rows: int64; bare: bool = false )

Queue PRE-ENCODED audio soft-token rows (what encode_audio emits — dim-wide, length-checked) for the next respond. MOVES rows; works on a plain chat, no tower needed. Pass the encoder’s audio_span_bare as bare: an ultravox span takes no marker, so a stock template serves it.

Arguments:

  • model : Model

  • chat : ChatSession

  • rows : array<float>

  • n_rows : int64

  • bare : bool

add_user_image(chat: ChatSession; img: VisionImage )

Queue an image for the next respond — geometry, letterbox and the embedder run NOW, spliced at the head of the next user turn. ONE image per turn; needs a chat created with create_chat(model, embedder); call inside with_job_que() (decode: dasllama_vision_io).

Arguments:
add_user_image_rows(model: Model; chat: ChatSession; rows: array<float>; n_rows: int64; grid: int2 )

Queue PRE-ENCODED image soft-token rows (what encode_image emits — deepstack models: (1+n)·dim-wide, length-checked) with the family’s mrope grid ((0,0) = sequential) for the next respond. MOVES rows; works on a plain chat — no embedder needed.

Arguments:

  • model : Model

  • chat : ChatSession

  • rows : array<float>

  • n_rows : int64

  • grid : int2

add_user_span(model: Model; chat: ChatSession; key: uint64; n_rows: int64; text_before: int; is_image: bool; bare: bool = false )

Mark the pending user turn as carrying a media span inline: render_turn and render_assistant then lay n_rows position ids (media_position_id over the media’s content key) where the media sits, text_before bytes into the turn’s text. The rows themselves are the caller’s to eval at those positions.

Arguments:

  • model : Model

  • chat : ChatSession

  • key : uint64

  • n_rows : int64

  • text_before : int

  • is_image : bool

  • bare : bool

17.1.10.1. create_chat

create_chat(model: Model; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): ChatSession

Start a conversation over model: resolves the chat template (GGUF-embedded, falling back to the arch registry) and creates the session. system is the system prompt (empty = none); max_new caps each reply; kv_dtype is the session’s KV-cache codec.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

  • kv_dtype : KVDtype

create_chat(model: Model; embedder: VisionEmbedder; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): ChatSession
create_chat(model: Model; tower: AudioTower; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): ChatSession
create_chat_renderer(model: Model; system: string = ""; max_new: int64 = 256 ): ChatSession

create_chat’s RENDER-ONLY twin: resolves the template/stop ids/turn close but creates NO KV session — a queued request can render its whole prompt holding tokens only, no cache memory. It cannot respond/eval.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

render_assistant(model: Model; chat: ChatSession; text: string; out: array<int64>; reasoning: string = "" )

add_assistant’s render half: appends the exact token stream a known reply prefills to out WITHOUT running the model, advancing the transcript like add_assistant (a create_chat_renderer chat replays history with no KV). reasoning renders in its think block under set_preserve_thinking.

Arguments:

  • model : Model

  • chat : ChatSession

  • text : string

  • out : array<int64>

  • reasoning : string

render_close(model: Model; chat: ChatSession ): array<int64>

The tokens that TERMINATE an assistant turn (what respond evals after the reply) — for schedulers that close a finished stream’s turn themselves.

Arguments:
render_turn(model: Model; chat: ChatSession ): array<int64>

Render the next turn’s prefill token ids — BOS + system on the first turn, then the user turn and the generation prompt — WITHOUT running the model. For inspection, token budgeting, tests.

Arguments:
render_turn_audio(model: Model; chat: ChatSession; head: array<int64>; tail: array<int64>; text_before: int = 0 )

render_turn’s AUDIO twin: the same two-span contract around the audio soft-token splice (the template’s audio span markers). Render-only (no KV, no encoder): the rows come from encode_audio, and a scheduler prefills head, rows, tail. text_before as in render_turn_image.

Arguments:

  • model : Model

  • chat : ChatSession

  • head : array<int64>

  • tail : array<int64>

  • text_before : int

render_turn_image(model: Model; chat: ChatSession; head: array<int64>; tail: array<int64>; text_before: int = 0 )

render_turn’s IMAGE twin: the two token spans that bracket the image soft-token splice — head before the rows, tail after. Render-only (no KV, no embedder): the rows come from encode_image. text_before bytes of the user text render ahead of the image (0: the image leads the turn).

Arguments:

  • model : Model

  • chat : ChatSession

  • head : array<int64>

  • tail : array<int64>

  • text_before : int

render_turn_marked(model: Model; chat: ChatSession ): RenderedTurn

render_turn with its opening marked: opening leading tokens - BOS, prelude and the system turn of a conversation’s first turn - are the same for any user text, so a prefix cache can checkpoint there.

Arguments:
render_user(model: Model; chat: ChatSession; out: array<int64> )

Render the pending user turn alone and consume it, with no generation prompt: the turn a user message that follows another user message makes, as every template writes them one after the other. Precondition: a user message pending; a no-op otherwise.

Arguments:
respond(model: Model; chat: ChatSession; params: SamplingParams; blk: block<(piece:string):bool> ): string

Generate the assistant’s reply to the queued user message, streaming pieces through the trailing block (return false to stop early). Terminates the turn in the KV cache and appends both turns to chat.history; returns the full reply text.

Arguments:
set_chat_date(date: string )

Pin the date a Llama-3.1+ system turn states ("26 Jul 2024"); "" returns it to the day the turn renders. For a prompt that must not move with the calendar - a pinned test stream, a benchmark.

Arguments:

  • date : string

set_thinking(chat: ChatSession; on: bool )

Toggle reasoning for a hybrid thinking model (Qwen3 family): false appends the template’s empty think block so the model answers directly. No-op without a suppress form or think specials in the vocab; default is on.

Arguments:

17.1.11. Tool calling

17.1.11.1. add_tool_results

add_tool_results(chat: ChatSession; results: array<string> )

Queue tool results as the next pending turn — the reply to an assistant turn that called tools. Call in place of add_user, then respond/render_turn as usual.

Arguments:
add_tool_results(chat: ChatSession; results: array<string>; names: array<string> )
parse_calls(chat: ChatSession; reply: string ): ToolReply

Parse a complete reply per the model family’s wire format into a ToolReply — the reasoning span, the content, and the calls with arguments normalized to JSON object text. A family with no tool format returns the reasoning/content split alone (safe on every reply); the buffered twin of the server’s streaming parse.

Arguments:
render_assistant_calls(model: Model; chat: ChatSession; text: string; calls: array<string>; out: array<int64>; reasoning: string = "" )

render_assistant’s tool-calling twin: replay an assistant turn that emitted tool calls (verbatim \{"name":…,"arguments":…} objects) plus any text alongside.

Arguments:

  • model : Model

  • chat : ChatSession

  • text : string

  • calls : array<string>

  • out : array<int64>

  • reasoning : string

set_tools(chat: ChatSession; tools: array<string> )

Declare the conversation’s tools (verbatim OpenAI tools[] JSON objects, moved in) BEFORE the first turn renders — the system turn carries the family’s tool block. Families with no tool format (tmpl.tool_mode none) ignore them.

Arguments:

17.1.12. Reasoning (thinking models)

effective_stop_ids(chat: ChatSession ): array<int64>

Every stop id for the NEXT generation as one list: the template’s stops plus the thinking-off extras while thinking is off. A scheduler that stops streams itself reads turn_stop_ids and runs the extras through nothink_stop_here instead.

Arguments:
make_nothink_guard(chat: ChatSession ): NothinkStopGuard

The instruct-mode stop guard for chat’s next turn over the family’s channel markers (gemma-4): unarmed on a thinking turn. Feed every sampled token through nothink_stop_here after the turn_stop_ids check.

Arguments:
make_think_stream(chat: ChatSession ): ThinkStream

The incremental reasoning/content splitter for chat’s next turn (think_feed per piece, think_finish at the end). Armed when the turn thinks, and on an instruct turn of a family with guarded channel markers (gemma-4); else a pass-through stream.

Arguments:
nothink_stop_here(guard: NothinkStopGuard; id: int64; piece: string ): bool

true = id ends the turn: a guarded channel marker sampled after the reply’s first content piece. A marker before content opens a leading thought the reply matcher splits.

Arguments:
set_preserve_thinking(chat: ChatSession; on: bool )

true makes every replayed assistant turn carry its think block - its reasoning between the template’s markers, the empty block when it had none - so a replayed transcript renders the tokens the model generated and a prefix cache matches through it. No-op where set_thinking is; default off.

Arguments:
set_reasoning_effort(chat: ChatSession; effort: string )

The reasoning level a harmony (gpt-oss) system turn asks for: low, medium or high. Unset, the level follows set_thinking: medium on, low off - the template has no off, and low is the shortest analysis it writes. No-op on every other family.

Arguments:
split_reasoning(chat: ChatSession; reply: string ): ThinkSplit

Split a complete reply at its reasoning boundary per the model family’s reply format (<think> pair, Harmony channels, gemma-4’s thought channel). Both halves come back stripped when a reasoning span is found; a reply with no reasoning passes through untouched.

Arguments:
think_budget_marks(model: Model; chat: ChatSession ): ThinkBudgetMarks

The token marks a reasoning budget acts on for chat’s next turn (ThinkBudgetMarks): the span’s open token, the close tokens a scheduler forces at a stream’s think_budget (the family’s close special; harmony’s end mark and final-channel header), and whether the prompt opened the span. Empty where the turn does not think.

Arguments:
think_drain(ts: ThinkStream; full: string ): ThinkSplit

Drain a COMPLETE reply through the splitter in one call: feed + finish + the strip rule (both halves strip when a reasoning span was consumed). The buffered-response twin of the think_feed/think_finish streaming pair.

Arguments:
think_feed(ts: ThinkStream; piece: string; reasoning: string&; content: string& )

Feed one streamed piece through the splitter; the out-strings are OVERWRITTEN with this piece’s reasoning/content deltas (either may be empty while a partial marker is held).

Arguments:

  • ts : ThinkStream

  • piece : string

  • reasoning : string&

  • content : string&

think_finish(ts: ThinkStream; reasoning: string&; content: string& )

Flush the splitter at end-of-generation (OVERWRITES the out-strings with the final deltas): an unclosed reasoning span classifies as reasoning — the truncated-tail rule.

Arguments:

  • ts : ThinkStream

  • reasoning : string&

  • content : string&

turn_stop_ids(chat: ChatSession ): array<int64>

The template’s own stop ids for the NEXT generation (<turn|>, <eos> and their kin): a token among them ends the turn outright. The thinking-off extras are the make_nothink_guard guard’s.

Arguments:

17.1.13. Operations: prepared images and dispatch

dlim_clean(gguf_path: string; apply: bool; keep_other: bool = false ): tuple<total:int;stale:int;removed:int;freed:int64>

Garbage-collect STALE images beside gguf_path and hash-named legacy images of another identity; a lane-named OTHER and every FOREIGN image stay (their own loads re-bake them in place). apply = false only reports, true removes; keep_other spares the legacy others too. Returns (total, stale, removed, freed bytes).

Arguments:

  • gguf_path : string

  • apply : bool

  • keep_other : bool

dlim_inventory(gguf_path: string ): array<DlimImageInfo>

List the prepared images (.dlim) minted beside gguf_path — per image: file, lane (the file name’s metal / cpu-<class> / vulkan / family segment, “” for a hash-named legacy image), bytes, version, identity, verdict (CURRENT loads; STALE vN an older version; OTHER another bake config or box; FOREIGN another flavor).

Arguments:

  • gguf_path : string

get_dispatch_worker_limit(): int

The dispatch worker cap in force (0 = no limit) — set_dispatch_worker_limit’s read half.

get_jobque_spin_gpu_us(): int64

The GPU-served spin window — set_jobque_spin_gpu_us’s read half.

get_jobque_spin_in_force(): int64

The spin window the pool runs right now: the GPU window while the whole-model driver serves the loaded model, the CPU window otherwise.

get_jobque_spin_us(): int64

The CPU spin window — set_jobque_spin_us’s read half. The window the pool runs at the moment is get_jobque_spin_in_force.

get_single_thread(): bool

Whether every kernel runs on the calling thread — set_single_thread’s read half.

kernel_backend_available(name: string ): bool

True when kernel backend name is registered AND its availability witness passes on this box — the detection probe behind defaults-first backend selection (the vulkan witness enumerates devices once and caches its verdict). Unknown names are false.

Arguments:

  • name : string

select_matmul_backend_for_load(): bool

Select the best matmul backend for the NEXT model load (honors a pin). Returns true when the choice needs the loader to repack weights into its interleaved layout.

set_dispatch_worker_limit(v: int )

Cap the kernel-dispatch worker count (0 = no limit, all job-que workers). Latched by setup_dasllama_jobque — set it before that call.

Arguments:

  • v : int

set_jobque_spin_gpu_us(v: int64 )

The workers’ spin-before-park window while the whole-model GPU driver serves the loaded model (default 500 us; a worker still spinning at the step’s submit starves the driver). A live queue takes it the moment the driver arms or drops; a later queue at its setup.

Arguments:

  • v : int64

set_jobque_spin_us(v: int64 )

The workers’ spin-before-park window in microseconds (0 = park at once). Latched by setup_dasllama_jobque — set it before that call; the default is 30 ms on every platform.

Arguments:

  • v : int64

set_single_thread(on: bool )

Run every kernel on the calling thread: no dispatch to the job queue at all, whatever queue exists. Takes effect at the next kernel, on THIS context (a new_thread body sets its own; DASLLAMA_SINGLE_THREAD=1 sets every context). The browser build’s baseline, and the figure that says what the pool buys on any box.

Arguments:

  • on : bool

17.1.14. Operations: GPU tier and model slots

gpu_model_marks_init(): GpuModelMarks

A fresh no-model marks holder — one per model slot; the marks save/restore pair swaps tier state across switches. (A generated constructor is not callable through the facade typedef, so the holder is minted here.)

gpu_slot_capture(st: GpuModelMarks ): string

Capture the INSTALLED model’s tier state into st right after its load_model and classify it — “gpu:resident” | “gpu:rails” | “cpu” — from the CAPTURED marks, never the process-global VRAM counter (it still holds the previous load’s bytes on a later CPU load).

Arguments:
gpu_slot_rearm(want: GpuTierWant; t: Model ): string

Re-arm the INSTALLED model onto the GPU tier — want, arm, resident upload (bake-slice path for a mapped vulkan-flavor t). Its marks are installed and the device holds no other model (moe_gpu_drop_model first). Returns “gpu:resident” | “gpu:rails” | “cpu”.

Arguments:
gpu_tier_status(): GpuTierStatus

Snapshot of the GPU tier in force: VRAM budget, resident/streamed layer counts, dense planes, engage/decline state — the server’s GPU badge reads this.

moe_gpu_drop_model()

Drop the INSTALLED model’s whole device state (resident stacks, mirrors, VRAM) — the evict half of a slot switch. Caller guarantees quiescence: no step in flight, every live session hydrated first.

moe_gpu_hydrate_session(t: Model; s: Session )

Pull s’s host KV back from the device mirror when it owns live mirror rows (no-op otherwise). Run over every live session of the outgoing model BEFORE moe_gpu_drop_model — the drop destroys the mirror.

Arguments:
moe_gpu_model_marks_restore(st: GpuModelMarks )

Install st as the per-model GPU tier state — moe_gpu_model_marks_save’s inverse; st reads as no-model after.

Arguments:
moe_gpu_model_marks_save(st: GpuModelMarks )

Save the INSTALLED model’s GPU tier state into st and disarm it — one half of the multi-model slot switch (the engine’s tier state is per-process, not per-model).

Arguments:
moe_gpu_tier_arm(): bool

Arm the recorded tier want so a GPU backend can install its hooks — call between set_gpu_tier_want and load_model. False when no backend claims the request.

moe_gpu_weight_budget(): int64

The armed backend’s resident-weight VRAM budget in bytes (0 = no GPU backend armed). CAVEAT: the hook device-inits on first call — a reporting caller gates on gpu_tier_status().device being non-empty first (supported must not be the gate).

resident_prefill_allowed(): bool

The pin as set_resident_prefill_allowed left it. A fixture reads it before it pins, then puts it back.

set_gpu_tier_want(want: GpuTierWant )

Record the GPU tier request the NEXT moe_gpu_tier_arm/load_model honors — the programmatic form of the core DASLLAMA_GPU_* knobs (an env var present overrides its field; the classifier/dense-arm knobs are env-only). A zero want keeps the load off the device.

Arguments:
set_resident_prefill_allowed(allowed: bool )

Allow or pin out the resident prefill arm for HOST-CACHED sessions: more of them than the driver has K/V regions would strand device-only rows. A device-home session prefills on the device whatever the pin says.

Arguments:

  • allowed : bool

17.1.15. Operations: serving from the device

create_device_session(model: Model; kv_dtype: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): Session

Create a DEVICE-HOME session: its K/V cache lives only in the armed GPU driver’s region - no host cache, nothing over the bus per token. Hold at most gpu_device_sessions() live ones, in the driver’s codec (gpu_device_session_dtype). A call the device cannot serve panics.

Arguments:
device_kv_max_rows(model: Model; kdt: KVDtype = dasllama_kv_dtype::KVDtype.f16; vdt: KVDtype = dasllama_kv_dtype::KVDtype.f16 ): int64

The rows of context the armed GPU driver holds a session’s K/V for under these codecs - the model’s own context where no driver bounds it. A server sizes its context by it: a blob-only model’s session past it has no CPU stack to fall back to.

Arguments:
gpu_cpu_passes(): array<tuple<reason:string;words:string;count:int64>>

Calls the armed GPU path handed back to the CPU rails since the model armed, one row per reason that fired: its name, the same in words a status page prints, the count. Empty means the device served every call.

gpu_ctx_pins(): tuple<ctx_max:int64;strict:bool>

The in-process context pins as set (set_gpu_ctx_max, set_gpu_ctx_strict), apart from the environment’s knobs: a fixture reads them before it pins and puts them back after its load.

gpu_device_kv_adopt(session: Session; claim: int64; npos: int64 ): bool

Hand the first npos rows a kept claim holds to the fresh device-home session, whose n_past becomes npos. False when the rows are gone (another session took the region).

Arguments:

  • session : Session

  • claim : int64

  • npos : int64

gpu_device_kv_park(session: Session ): int64

A FINISHED device-home session lets go of its K/V region and answers the claim its rows stay under (0 = none) - call it before every such session’s delete. Keep the claim and gpu_device_kv_adopt hands the rows to a later session: a prefix cache with no copy.

Arguments:
gpu_device_prefill_continues(model: Model ): bool

A device-home session of model prefills from a position past zero - a prompt in chunks, a turn over the turns before it. False for a recurrent model: prefill each prompt whole, from position zero, and adopt no earlier rows.

Arguments:
gpu_device_prefill_window(): int64

The positions the device prefills in one window - the shortest chunk worth handing a device-home session.

gpu_device_session_ctx(): int64

The positions one device-home session holds - the armed driver’s per-region context.

gpu_device_session_dtype(): KVDtype

The K/V codec a device-home session must carry: the armed driver’s.

gpu_device_sessions(): int64

How many DEVICE-HOME sessions (create_device_session) the installed model serves at once: the armed driver’s region count, 0 when the model is not served whole from the device.

gpu_dn_room(n: int64 )

Tell the GPU driver how many sessions’ recurrent state it keeps device-resident at once: a scheduler names its stream count before the first prefill, so no stream’s deltanet state evicts under the batched step it rides. A driver with no such cache ignores it.

Arguments:

  • n : int64

gpu_kv_dtype_pin(): tuple<pinned:bool;dt:KVDtype>

The in-process ask as set_gpu_kv_dtype left it: pinned false where no caller asked and DASLLAMA_GPU_KV decides. A fixture reads it before it pins, then puts it back with set_gpu_kv_dtype or reset_gpu_kv_dtype.

gpu_resident_decline(): string

Why the whole-model GPU driver does not serve the model loaded last: its decline reason (with the remedy where one exists), or why it was never attempted. Empty when it serves.

gpu_resident_experts_host(): bool

Whether the whole-model driver serving the model loaded last sums some routed layers’ experts on the host (a hyper-connection MoE, or a plain MoE the card does not hold whole) - the form whose speculative round does not pay, so a server slot leaves the round off unless asked.

reset_gpu_kv_dtype()

Lift the in-process ask set_gpu_kv_dtype made: DASLLAMA_GPU_KV decides the next load’s mirror codec again.

set_gpu_ctx_max(n: int64 )

Cap the whole-model GPU driver’s K/V context at n positions per region for the models loaded next (0 lifts it): a server’s ctx. The card’s room still bounds it; a cap under the arming floor lowers the floor to the cap (never under 256): the cap is the shape the caller serves.

Arguments:

  • n : int64

set_gpu_ctx_strict(on: bool )

The set_gpu_ctx_max pin is the context asked, not a cap, for the models loaded next: a K/V mirror the card cannot hold at it declines the whole-model GPU driver instead of shortening the context, and the per-op rails serve the asked context (DASLLAMA_GPU_CTX_STRICT in the environment).

Arguments:

  • on : bool

set_gpu_kv_dtype(dt: KVDtype )

Ask the whole-model GPU driver to hold its K/V mirror in codec dt for the models loaded next - the codec of their sessions (gpu_device_session_dtype reads what armed). q8_0 is about half the default f16 mirror’s bytes, tq4 about a quarter; f32 reads as f16, and so does a model whose heads the codec cannot hold.

Arguments:
set_gpu_resident_regions(n: int64 )

Ask the whole-model GPU driver for n K/V regions on the models loaded next, so n sessions keep their K/V on the device at once - a server’s stream count. One by default.

Arguments:

  • n : int64