8.7.8. dasLLAMA-08 — Audio Chat

Audio-capable chat models pair a normal text decoder with a whisper-family audio encoder — the tower — that turns 16 kHz PCM into soft tokens the decoder reads inline with text. Supported pairs (decoder + mmproj GGUF): Qwen2-Audio, Qwen2.5-Omni (audio side), Ultravox v0.5 (over stock Llama-3 decoders), and Voxtral-Mini. The chat template picks the audio framing automatically — the code below is identical for every pair.

Run:

daslang.exe -jit tutorials/dasLLAMA/08_audio_chat.das -- decoder.gguf mmproj.gguf clip.wav

8.7.8.1. A chat that hears

create_chat(model, tower) moves the tower into the chat (which owns the encoder scratch too — one chat, one hearing apparatus). add_user_audio encodes the clip to soft tokens immediately and queues them at the head of the next turn; add_user contributes the turn’s text after the audio span. respond renders the turn — template framing, audio splice, embedding prefill — and streams the reply.

var m <- load_model("Llama-3.2-1B-Instruct-Q8_0.gguf", QuantMode.q8)
var tower <- load_audio_tower("mmproj-ultravox-1b-f32.gguf")

var chat <- create_chat(m, tower)       // panics on a mismatched pair
add_user_audio(chat, samples)           // 16 kHz mono f32 PCM
add_user(chat, "What is being said in this audio?")
respond(m, chat, SamplingParams()) $(piece) {
    print("{piece}")
    return true
}

Follow-up turns work like any text chat — the audio turn is in the KV cache, so the model remembers what it heard. A later add_user_audio call brings a second clip into a new turn.

8.7.8.2. What the template does with audio

Each model family frames the audio span its own way, and the chat layer wires it from the model’s template: the qwen2 family wraps the soft tokens in <|audio_bos|> / <|audio_eos|>, Llama-3-based Ultravox splices bare embeddings, Voxtral opens the span with [BEGIN_AUDIO]. Text runs break at the audio boundary, so tokenizer merges never cross it — the rendered turn is token-identical to llama.cpp’s mtmd reference for every family.

8.7.8.3. caps() for chat models

caps(model) returns LlmCaps — the chat layer’s honesty contract. Its first entry exists because gemma has no system role: the chat layer folds the system prompt into the first user turn, and system_prompt == false is how a program finds out instead of being silently absorbed.

if (!caps(m).system_prompt) {
    print("note: this model has no system role — the prompt is folded into turn 1\n")
}