8.7.15. dasLLAMA-14 — Vision Chat

Tutorial 08 gave a conversation ears. This one gives it eyes. A vision chat model pairs a normal text decoder with an image encoder: the encoder turns pixels into soft tokens — embedding rows the decoder reads inline with text, one row per group of image patches. The supported families are gemma-4 dense (the gemma4uv embedder), gemma-4 E-series (the gemma4v ViT tower), and gemma-3 (the gemma3v SigLIP tower), each shipped as a decoder GGUF plus its vision multimodal projector (mmproj) GGUF; the loader sniffs which family an mmproj is.

Run it with a decoder, its mmproj, and any image stbimage decodes:

daslang.exe -jit tutorials/dasLLAMA/14_vision_chat.das -- decoder.gguf mmproj.gguf photo.jpg "What is in this picture?"

8.7.15.1. A conversation that can see

The easy path is four calls. load_image_rgb decodes the file to an RGB8 VisionImage. load_vision_embedder reads the mmproj — the vision half of the model. create_chat(model, embedder) moves the embedder into the chat, and add_user_image queues the picture for the next turn: geometry, letterbox and the encoder run right there, so respond only prefills and generates. One image per turn; text via add_user rides along:

var inscope img <- load_image_rgb(image_path)
var emb <- load_vision_embedder(mmproj_path)
var chat <- create_chat(m, emb, "", 96l)
add_user_image(chat, img)
add_user(chat, "Describe this image in one sentence.")
respond(m, chat, SamplingParams()) $(piece) {
    print("{piece}")
    return true
}

8.7.15.2. Soft tokens by hand

Now the same turn built ourselves — the shape a scheduler uses when the encoder runs on a worker and the decoder session lives elsewhere. encode_image is the whole image path in one call: geometry, letterbox, normalize, encode. It appends npos rows of model.config.dim floats and returns npos. render_turn_image renders the turn’s tokens as two spans — head before the image rows, tail after — split exactly at the template’s image marker, so tokenizer merges never cross the picture:

var scratch = VisionState()
var img_rows : array<float>
let n_img = encode_image(emb, scratch, img, "tag", img_rows)

var chat = create_chat_renderer(m, "", 96l)
add_user(chat, prompt)
var head : array<int64>
var tail : array<int64>
render_turn_image(m, chat, head, tail)

The prompt we prefill is head rows + image rows + tail rows, one array — prefilling tokens is prefilling their embedding rows (tutorial 08 proves that identity), so embed_text_rows fills the text parts. Then eval_embd_span runs the whole thing with one twist: the rows in [span_lo, span_hi) are non-causal — every query in the span attends the whole span, because a picture has no left-to-right order. The text around it stays causal:

eval_embd_span(m, s, rows, np, n_head, n_head + n_img)
// ...then sample() one token at a time, like any session

eval_embd is the same call without a span — every row causal. Audio rows keep their time order, so the audio family prefills through it; use it whenever you already hold the embedding rows and no image span is in play.

The qwen families (Qwen3-Omni, Qwen3-VL, Qwen2.5-Omni) rope image rows by their 2D patch position instead of by sequence order. After the encode, vision_mrope_grid reports the merged grid, and eval_embd_span_mrope prefills the same non-causal span with grid-shaped angles — the position advance after the image is max(grid.x, grid.y), not the row count, and the session tracks that delta for every later eval:

let grid = vision_mrope_grid(emb, scratch)
eval_embd_span_mrope(m, s, rows, np, n_head, n_head + n_img, grid)

A gemma pair reports a zero grid — the plain eval_embd_span shape serves it, and the tutorial’s third section falls back exactly that way. The same section then runs the one-call form: generate_embd takes the spliced rows, the span bounds and the grid, prefills and streams the reply — what respond runs under the hood for a media turn. The section closes with the pre-encoded-rows seam: add_user_image_rows moves the same soft tokens and the grid onto a plain chat — no embedder attached — and respond then runs the spliced turn. That is the path for a scheduler that owns its own embedder:

var chat2 = create_chat(m, "", 96l)
add_user_image_rows(m, chat2, img_rows, n_img, grid)
add_user(chat2, prompt)

One boundary worth knowing: a deepstack pair (dense Qwen3-VL) encodes wider rows — (1 + n_deepstack) × dim floats each; the narrow by-hand splice still captions (the extra slices are additive refinement), and create_chat(model, embedder) or the add_user_image_rows seam are the paths that carry them in full — the seam length-checks the wide quantum.