8.7.14. dasLLAMA-13 — Serving Many Conversations

Every tutorial so far drove one conversation at a time. A server holds many — and dasLLAMA’s serving layer is the same facade plus one module, dasllama/dasllama_scheduler: the continuous-batching scheduler behind dasllama-server. This capstone shows the raw mechanism once, then drives the real thing.

Run it like tutorial 01 (SmolLM2-135M-Instruct works well):

daslang.exe -jit tutorials/dasLLAMA/13_serving.das -- path/to/model.gguf

8.7.14.1. The raw mechanism

Serving B conversations means B sessions over one model, and two primitives make that cheap. A paged KV pool: sessions created over create_kv_pool allocate fixed-size page groups on demand instead of a full seq_len slab each (tutorial 04 sized one such slab). And eval_batch: one decode step for all sessions in one pass of the weights — B matrix-vector products fuse into one matrix-matrix product, which is where the throughput comes from:

var pool <- create_kv_pool(m, 16l)
var ws <- create_batch_workspace(m)
var s0 = create_session(m, pool)
var s1 = create_session(m, pool)
var rows <- [unsafe(addr(s0)), unsafe(addr(s1))]
var toks <- [sample(*rows[0], SamplingParams()), sample(*rows[1], SamplingParams())]
eval_batch(m, ws, rows, toks)   // ONE weight pass advances every stream

8.7.14.2. The scheduler

dasllama_scheduler is that step made production-shaped. Requests are rendered to tokens with no session — create_chat_renderer supplies the template, stop ids, and turn close, so a deep queue holds no KV memory:

var chat = create_chat_renderer(m, "You are a concise assistant.", 32l)
add_user(chat, "What is the capital of France?")
var req <- PendingReq(id = 1l, max_new = 32l)
req.prompt <- render_turn(m, chat)
req.close_toks <- render_close(m, chat)
req.stop_ids <- effective_stop_ids(chat)

create_scheduler sizes the machine; submit queues requests; scheduler_step admits, runs one batched decode step over every live stream, then at most one bounded prefill chunk — a long prompt stalls the others by a chunk, not by its whole length. Results come back as SchedEvent values:

var sch <- create_scheduler(m, 4l, 32l, 64l, 16l)   // 4 streams, paged KV, prefix cache
submit(m, sch, req)
var events : array<SchedEvent>
while (scheduler_step(m, sch, events)) {
    for (ev in events) {
        if (ev.kind == SchedEventKind.piece) {
            print("{ev.text}")
        }
    }
    events |> clear()
}

The tutorial prints one digit — the stream id — per generated piece, so the run shows continuous batching literally:

123412341234 234234234 3434343434...

Streams finish at their own pace; new ones admit while others still decode. On four SmolLM2 conversations the batched round beats the same four run one at a time by roughly a third of the wall clock — and the gap grows with stream count and model size.

8.7.14.3. The prefix cache

With page_rows set, a finished stream donates its KV pages to a prefix cache. The four prompts share one system turn, so a second round of new questions attaches the shared pages instead of prefilling them — the finished event reports it:

[warm stream 2] stop: 10 tok, ttft 22ms, cached 48 of 67 prompt tokens

Forty-eight of sixty-seven prompt tokens — the system turn’s three pages — arrived from the cache; only the question’s tail paid prefill, and time to first token dropped to match.

8.7.14.4. The same module, more streams

That is the whole serving story: rendered stateless requests, one batched step, pages donated and reattached. The OpenAI-compatible server (utils/dasllama-server) drives this exact scheduler under HTTP and SSE — same submit, same scheduler_step, same events.