8.7.15. dasLLAMA-14 — Vision Chat

Tutorial 08 gave a conversation ears. This one gives it eyes. A vision chat model pairs a normal text decoder with an image encoder: the encoder turns pixels into soft tokens — embedding rows the decoder reads inline with text, one row per group of image patches. The supported family is gemma-4 dense, shipped as a decoder GGUF plus its vision multimodal projector (mmproj) GGUF.

Run it with a decoder, its mmproj, and any image stbimage decodes:

daslang.exe -jit tutorials/dasLLAMA/14_vision_chat.das -- decoder.gguf mmproj.gguf photo.jpg "What is in this picture?"

8.7.15.1. A conversation that can see

The easy path is four calls. load_image_rgb decodes the file to an RGB8 VisionImage. load_gemma4uv_embedder reads the mmproj — the vision half of the model. create_chat(model, embedder) moves the embedder into the chat, and add_user_image queues the picture for the next turn: geometry, letterbox and the encoder run right there, so respond only prefills and generates. One image per turn; text via add_user rides along:

var inscope img <- load_image_rgb(image_path)
var emb <- load_gemma4uv_embedder(mmproj_path)
var chat <- create_chat(m, emb, "", 96l)
add_user_image(chat, img)
add_user(chat, "Describe this image in one sentence.")
respond(m, chat, SamplingParams()) $(piece) {
    print("{piece}")
    return true
}

8.7.15.2. Soft tokens by hand

Now the same turn built ourselves — the shape a scheduler uses when the encoder runs on a worker and the decoder session lives elsewhere. encode_image is the whole image path in one call: geometry, letterbox, normalize, encode. It appends npos rows of model.config.dim floats and returns npos. render_turn_image renders the turn’s tokens as two spans — head before the image rows, tail after — split exactly at the template’s image marker, so tokenizer merges never cross the picture:

var scratch = Gemma4uvState()
var img_rows : array<float>
let n_img = encode_image(emb, scratch, img, "tag", img_rows)

var chat = create_chat_renderer(m, "", 96l)
add_user(chat, prompt)
var head : array<int64>
var tail : array<int64>
render_turn_image(m, chat, head, tail)

The prompt we prefill is head rows + image rows + tail rows, one array — prefilling tokens is prefilling their embedding rows (tutorial 08 proves that identity), so embed_text_rows fills the text parts. Then eval_embd_span runs the whole thing with one twist: the rows in [span_lo, span_hi) are non-causal — every query in the span attends the whole span, because a picture has no left-to-right order. The text around it stays causal:

eval_embd_span(m, s, rows, np, n_head, n_head + n_img)
// ...then sample() one token at a time, like any session

eval_embd is the same call without a span — every row causal. Audio rows keep their time order, so the audio family prefills through it; use it whenever you already hold the embedding rows and no image span is in play.