17.1. dasLLAMA LLM inference: models, sessions, generation, chat

CPU large-language-model inference in pure daslang: load a GGUF model, tokenize, run the transformer, sample — or hold a full chat — validated token-for-token against llama.cpp on every supported family. Run with -jit; examples/dasLLAMA/run.das and chat.das show the canonical program shape.

Supported model families (GGUF — fp32 / f16 / q8_0 / q4_0 / mxfp4 weights read directly; K-quant files such as Q4_K_M / Q5_K_M / Q6_K run on native K-quant kernels):

  • Llama — Llama-2 / TinyLlama, Llama-3.1 / 3.2, Mistral-7B-Instruct, SmolLM2, plus llama2.c .bin checkpoints

  • Qwen — Qwen2.5, Qwen3 (QK-norm), Qwen3.5 / Qwen3.6 (hybrid Gated-DeltaNet attention, incl. the 35B-A3B MoE); MoE: Qwen1.5-MoE (routed + sigmoid-gated shared expert), Qwen3-30B-A3B (routed-only, renormalized top-k)

  • Phi — Phi-3.5-mini

  • Gemma — Gemma-2, Gemma-3 (per-layer sliding-window patterns), Gemma-4 (12B / 31B dense, the 26B-A4B MoE, and the E2B / E4B edge series with per-layer embeddings + cross-layer KV sharing)

  • gpt-oss — gpt-oss-20b (attention sinks, native MXFP4 experts, YaRN long context, Harmony chat format)

The architecture is picked from GGUF metadata at load — the same program runs any of these.

Hands-on tutorials (overview): the problem statement, hello, generation, chat and templates, sampling, sessions and memory, performance, the architecture registry.

17.1.1. Types

The engine types the API below works with. They are created and consumed by the functions of this module; their remaining fields are engine implementation detail.

Model

A loaded model: weights, config, tokenizer, and the architecture’s blocks and chat template, as produced by load_model. User code touches config (e.g. cap config.seq_len before create_session on large-context models) and arch (the GGUF architecture name).

Session

One generation stream over a model: the KV cache, scratch buffers, sampling RNG, and the current position n_past. logits holds the distribution produced by the last eval. A model serves many independent sessions.

BatchWorkspace

Caller-owned scratch for eval_batch: the batched activation buffers a step of B sessions shares. One per concurrent batch, reused across calls; holds no session state — positions, caches and logits stay in the sessions.

KVPool

A caller-owned paged KV-cache pool (create_kv_pool): sessions created over it allocate fixed-size page groups on demand, so cache memory tracks the actual context instead of the full seq_len slab. One pool serves many sessions; an eval_batch batch must share one pool. Keep it alive and in place while its sessions live.

PrefixCache

A page-granular prefix cache over one pool’s sessions (create_prefix_cache): finished streams donate their KV pages keyed by a chained page hash of the token history, and later requests attach the longest cached prefix instead of re-prefilling it. Pages are refcounted with the pool; an LRU budget bounds retention.

KVDtype

Per-session KV-cache codec picked at create_session/create_chat: f16 — the default, half the KV bytes and faster deep-context decode (stores clamp to ±65504); f32 — the bit-exact reference; q8_0 — block-quantized (llama.cpp -ctk/-ctv q8_0), half the f16 bytes again and near-lossless in practice; needs head_size and kv_dim to be multiples of 32 (checked at create); tq4 — the FWHT-rotated ternary codec, smaller still.

QuantMode

Weight representation picked at load: fp32 — the token-exact reference; q8 — int8 quantization, the fast CPU path (K-quant/mxfp4/Q4_0 files keep their native planes on the same rails); q4_0 — the legacy requant tier, 4-bit blocks, smallest footprint.

SamplingParams

Sampling knobs: temp (<= 0 selects greedy argmax), top_k (0 = no cutoff), and repetition penalty (1.0 = none) applied over the last penalty_last_n generated tokens. The defaults are greedy — SamplingParams() reproduces argmax exactly.

Stats

Timing of the last generate/respond call: n_prompt/n_gen token counts, ttft_s (seconds to first token), and prefill_tps/gen_tps throughput in tokens per second.

LlmCaps

What the model honestly supports at the chat layer, as returned by caps: system_prompt is false for architectures with no system role (gemma), where the chat layer silently folds the system prompt into the first user turn. Grows as gaps surface.

ChatSession

A conversation over a model: its session, the resolved chat template, and the running transcript in history. Create with create_chat, then drive with add_user + respond.

ThinkStream

The incremental reasoning/content splitter for a streamed reply (make_think_stream): feed decoded pieces through think_feed, flush with think_finish. Holds partial reasoning markers back across chunk boundaries; a non-thinking family yields a pass-through stream.

ThinkSplit

One reply split at its reasoning boundary, as returned by split_reasoning: reasoning holds the thinking span (empty when the model answered directly), content the answer.

ToolReply

A fully classified tool-capable reply, as returned by parse_calls: the reasoning span, the content, and the calls the model made (empty when it answered directly).

ToolCall

One parsed tool call: the function name, its args normalized to JSON object text, and the wire id where the family carries one (mistral; empty elsewhere).

AudioTower

A loaded audio encoder: Whisper-family encoder weights plus the model-specific projector tail, as produced by load_audio_tower from an mmproj GGUF. Pass it to create_chat to enable add_user_audio turns.

17.1.2. Model loading and sessions

caps(model: Model ): LlmCaps

What model honestly supports at the chat layer (see LlmCaps) — e.g. gemma has no system role, so the chat layer folds the system prompt into the first user turn; system_prompt is false there so callers can surface it instead of being silently absorbed.

Arguments:
create_batch_workspace(model: Model ): BatchWorkspace

Create the caller-owned scratch that eval_batch steps through — one per concurrent batch, reused across calls (buffers grow to the largest batch seen). Holds no session state: the sessions keep their own positions, caches and logits.

Arguments:
create_kv_pool(model: Model; page_rows: int64 = 64; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): KVPool

Create a caller-owned PAGED KV pool over model’s cache geometry. Sessions created over it allocate cache pages of page_rows positions on demand instead of the full seq_len slab up front, so many sessions share one elastic pool. Keep the pool alive as long as sessions live.

Arguments:

17.1.2.1. create_session

create_session(model: Model; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): Session

Create a fresh session (KV cache + scratch) sized to model.config.seq_len — one model, many independent conversations. kv_dtype picks the KV-cache codec: f16 (default, near-lossless) halves bytes, q8_0 halves again (needs head_size/kv_dim % 32 == 0).

Arguments:
create_session(model: Model; pool: KVPool ): Session
load_model(path: string; mode: QuantMode = dasllama_common::QuantMode.fp32 ): Model

Load a model AND its tokenizer from a GGUF file — architecture and tokenizer backend are auto-selected from metadata; mode picks the weight quantization. Q8 loads cache a PREPARED IMAGE beside the gguf for millisecond reloads (DASLLAMA_IMAGE=0 disables); under an active Metal mode the image is the BLOB-ONLY metal flavor (CPU inference against it panics).

Arguments:
release_kv_pages(session: Session )

Return session’s KV pages to its pool (no-op on flat sessions). The normal shape is release + delete; a released session stays alive but loses its cached context — to reuse it, also reset session.n_past to 0.

Arguments:
setup_dasllama_jobque()

Configure the job queue for dasLLAMA’s fork/join matmul dispatch: pooled fork contexts, batched dispatch, and the worker spin-before-park window (jobque_spin_us; 0 disables). Call INSIDE with_job_que(), before the first generate/eval.

17.1.3. Prefix cache

create_prefix_cache(max_groups: int64 = 0 ): PrefixCache

Create a prefix cache for the paged sessions of one create_kv_pool pool: finished streams donate KV pages (prefix_insert), later requests with the same prefix attach them (prefix_attach) instead of re-prefilling. max_groups caps retained groups (0 = unbounded).

Arguments:

  • max_groups : int64

prefix_attach(cache: PrefixCache; pool: KVPool; session: Session; prompt: array<int64> ): int64

Attach the longest cached prefix of prompt to a FRESH paged session of pool: matched pages join the session’s block table and n_past advances past them, so the caller prefills only the tail. Returns the matched count, capped one token short of the prompt.

Arguments:
prefix_chain_list(cache: PrefixCache ): array<PrefixChain>

Snapshot of the cache’s donated chains for dashboards: per donation — page-covered token count, live pages, hit count, born/last-hit ticks, and the caller-provided preview.

Arguments:
prefix_held_groups(cache: PrefixCache ): int64

Pages the cache currently holds (== pool groups retained for reuse).

Arguments:
prefix_insert(cache: PrefixCache; pool: KVPool; session: Session; tokens: array<int64>; preview: string = "" )

Donate a finished session’s KV pages to the cache. tokens is the session’s full EVALED history (only the first n_past rows exist); every full page not already cached survives release_kv_pages. preview labels the chain on the prefix_chain_list surface.

Arguments:
prefix_release(cache: PrefixCache; pool: KVPool )

Release every cached page back to pool and clear the cache (pages still used by live sessions stay alive until those sessions release them). Call before deleting the pool.

Arguments:

17.1.4. Tokenizer

decode(model: Model; ids: array<int64> ): string

Decode a token-id sequence back to text with the model’s tokenizer.

Arguments:

  • model : Model

  • ids : array<int64>

encode(model: Model; text: string; add_special: bool = true; parse_special: bool = false ): array<int64>

Encode text to token ids with the model’s tokenizer. add_special prepends BOS where the model expects one. parse_special is reserved and not yet honored — special tokens reach the model as atomic ids from the chat layer’s template renderer, never by spelling them in text.

Arguments:

  • model : Model

  • text : string

  • add_special : bool

  • parse_special : bool

piece(model: Model; id: int64 ): string

Decode a single token to its text piece — the streaming counterpart of decode.

Arguments:

  • model : Model

  • id : int64

17.1.5. Evaluation and sampling

eval(model: Model; session: Session; tokens: array<int64> )

THE eval primitive: run tokens at the session’s current position and advance it. Prefill = eval(prompt); each generation step = eval([token]) — the same call at different batch sizes. Logits land in session.logits.

Arguments:
eval_batch(model: Model; ws: BatchWorkspace; sessions: array<Session?>; tokens: array<int64> )

One synchronous batched decode step: row i evals tokens[i] at sessions[i]’s current position, advancing each by one — B conversations through ONE pass of the weights (GEMVs batch into GEMMs). Sessions must be distinct, same-geometry, one pool if paged.

Arguments:
eval_embd(model: Model; session: Session; embd: array<float>; npos: int64 )

eval’s embedding-input twin: prefill npos pre-built embedding rows (npos × dim, token-major) at the session’s current position and advance it — the multimodal splice entry (text via embed_text_rows, media via an encoder tower’s soft tokens).

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

sample(session: Session; params: SamplingParams ): int64

Sample the next token from session.logits per params: penalties, then temperature/top-k/top-p/min-p and a CDF draw — or greedy argmax when params.temp <= 0 (SamplingParams() defaults are greedy).

Arguments:
set_seed(session: Session; seed: int )

Seed the session’s sampling RNG for reproducible generation.

Arguments:
stats(session: Session ): Stats

Timing of the most recent generate/respond call on session: prompt/generated token counts, time to first token, prefill and generation tok/s.

Arguments:

17.1.6. Generation

generate(model: Model; session: Session; prompt: array<int64>; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

Stream-generate up to max_tokens from prompt, invoking the trailing block per token with (id, piece); return false from the block to stop early. Prefills the prompt in one eval, then samples one token at a time; returns the number of tokens emitted.

Arguments:

  • model : Model

  • session : Session

  • prompt : array<int64>

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

generate_embd(model: Model; session: Session; embd: array<float>; npos: int64; params: SamplingParams; max_tokens: int64; blk: block<(id:int64;piece:string):bool> ): int64

generate’s embedding-prefill twin: prefill npos pre-built embedding rows (the multimodal splice — see eval_embd), then stream-sample exactly like generate. The chat layer’s audio turns run on this; use it directly for custom multimodal prompts.

Arguments:

  • model : Model

  • session : Session

  • embd : array<float>

  • npos : int64

  • params : SamplingParams

  • max_tokens : int64

  • blk : block<(id:int64;piece:string):bool>

17.1.7. Embeddings

embed(model: Model; text: string ): array<float>

Mean-pooled, L2-normalized sentence embedding of text (model.config.dim floats): the decoder’s last-layer hidden state (post-final RMSNorm), averaged then unit-normalized. A decoder-only model used this way yields RAG-grade vectors, not a dedicated embedder’s.

Arguments:

  • model : Model

  • text : string

17.1.8. Chat

add_assistant(model: Model; chat: ChatSession; text: string )

Inject a KNOWN assistant reply (no generation): prefill the pending user turn and text into the KV cache, then close the turn — like respond but with a supplied reply. The shape a stateless server needs to replay history. Precondition: a user message is pending.

Arguments:
add_user(chat: ChatSession; text: string )

Queue a user message for the next respond.

Arguments:
add_user_audio(chat: ChatSession; samples: array<float>|array<float># ): auto

Queue audio (16 kHz mono f32 PCM) for the next respond — encoded to soft tokens immediately and spliced at the head of the turn before any add_user text. Needs a chat created with create_chat(model, tower); call inside with_job_que().

Arguments:

  • chat : ChatSession

  • samples : option<array<float>| array<float>#>

17.1.8.1. create_chat

create_chat(model: Model; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): ChatSession

Start a conversation over model: resolves the chat template (GGUF-embedded, falling back to the arch registry) and creates the session. system is the system prompt (empty = none); max_new caps each reply; kv_dtype is the session’s KV-cache codec.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

  • kv_dtype : KVDtype

create_chat(model: Model; tower: AudioTower; system: string = ""; max_new: int64 = 256; kv_dtype: KVDtype = dasllama_common::KVDtype.f16 ): ChatSession
create_chat_renderer(model: Model; system: string = ""; max_new: int64 = 256 ): ChatSession

create_chat’s RENDER-ONLY twin: resolves the template/stop ids/turn close but creates NO KV session — a queued request can render its whole prompt holding tokens only, no cache memory. It cannot respond/eval.

Arguments:

  • model : Model

  • system : string

  • max_new : int64

render_assistant(model: Model; chat: ChatSession; text: string; out: array<int64> )

add_assistant’s render half: appends the exact token stream a known reply prefills to out WITHOUT running the model, advancing the transcript like add_assistant. Use on a create_chat_renderer chat to replay history with no KV memory. Precondition: user message pending.

Arguments:
render_close(model: Model; chat: ChatSession ): array<int64>

The tokens that TERMINATE an assistant turn (what respond evals after the reply) — for schedulers that close a finished stream’s turn themselves.

Arguments:
render_turn(model: Model; chat: ChatSession ): array<int64>

Render the next turn’s prefill token ids — BOS + system on the first turn, then the user turn and the generation prompt — WITHOUT running the model. For inspection, token budgeting, tests.

Arguments:
respond(model: Model; chat: ChatSession; params: SamplingParams; blk: block<(piece:string):bool> ): string

Generate the assistant’s reply to the queued user message, streaming pieces through the trailing block (return false to stop early). Terminates the turn in the KV cache and appends both turns to chat.history; returns the full reply text.

Arguments:
set_thinking(chat: ChatSession; on: bool )

Toggle reasoning for a hybrid thinking model (Qwen3 family): false appends the template’s empty think block so the model answers directly. No-op without a suppress form or think specials in the vocab; default is on.

Arguments:

17.1.9. Tool calling

17.1.9.1. add_tool_results

add_tool_results(chat: ChatSession; results: array<string> )

Queue tool results as the next pending turn — the reply to an assistant turn that called tools. Call in place of add_user, then respond/render_turn as usual.

Arguments:
add_tool_results(chat: ChatSession; results: array<string>; names: array<string> )
parse_calls(chat: ChatSession; reply: string ): ToolReply

Parse a complete reply per the model family’s wire format into a ToolReply — the reasoning span, the content, and the calls with arguments normalized to JSON object text. A family with no tool format returns the reasoning/content split alone (safe on every reply); the buffered twin of the server’s streaming parse.

Arguments:
render_assistant_calls(model: Model; chat: ChatSession; text: string; calls: array<string>; out: array<int64> )

render_assistant’s tool-calling twin: replay an assistant turn that emitted tool calls (verbatim \{"name":…,"arguments":…} objects) plus any text alongside.

Arguments:

  • model : Model

  • chat : ChatSession

  • text : string

  • calls : array<string>

  • out : array<int64>

set_tools(chat: ChatSession; tools: array<string> )

Declare the conversation’s tools (verbatim OpenAI tools[] JSON objects, moved in) BEFORE the first turn renders — the system turn carries the family’s tool block. Families with no tool format (tmpl.tool_call_open empty) ignore them.

Arguments:

17.1.10. Reasoning (thinking models)

effective_stop_ids(chat: ChatSession ): array<int64>

The stop ids in force for the NEXT generation: the template’s stops plus its thinking-off extras while thinking is off. Schedulers that stop streams themselves read this, not chat.stop_ids, so an instruct-mode gemma-4 cuts at a stray channel marker.

Arguments:
make_think_stream(chat: ChatSession ): ThinkStream

The incremental reasoning/content splitter for chat’s next turn — feed streamed pieces through think_feed, flush with think_finish. Armed only when the turn actually thinks (toggle on, markers in the vocab, gate rendered) — else a pass-through stream.

Arguments:
split_reasoning(chat: ChatSession; reply: string ): ThinkSplit

Split a complete reply at its reasoning boundary per the model family’s reply format (<think> pair, Harmony channels, gemma-4’s thought channel). Both halves come back stripped when a reasoning span is found; a reply with no reasoning passes through untouched.

Arguments:
think_drain(ts: ThinkStream; full: string ): ThinkSplit

Drain a COMPLETE reply through the splitter in one call: feed + finish + the strip rule (both halves strip when a reasoning span was consumed). The buffered-response twin of the think_feed/think_finish streaming pair.

Arguments:
think_feed(ts: ThinkStream; piece: string; reasoning: string&; content: string& )

Feed one streamed piece through the splitter; the out-strings are OVERWRITTEN with this piece’s reasoning/content deltas (either may be empty while a partial marker is held).

Arguments:

  • ts : ThinkStream

  • piece : string

  • reasoning : string&

  • content : string&

think_finish(ts: ThinkStream; reasoning: string&; content: string& )

Flush the splitter at end-of-generation (OVERWRITES the out-strings with the final deltas): an unclosed reasoning span classifies as reasoning — the truncated-tail rule.

Arguments:

  • ts : ThinkStream

  • reasoning : string&

  • content : string&