8.7.17. dasLLAMA-16 — Text to Speech

dasLLAMA speaks through one small surface: a loader that reads the family from the GGUF, a caps() call that lists the voices, and two verbs — synthesize for the whole text at once, synthesize_stream for one sentence at a time. The same program runs KittenTTS (kitten-nano and kitten-mini), Kokoro-82M and Pocket TTS.

The first three are StyleTTS2-lineage models, and they share one assembly. The text becomes phonemes, a text encoder predicts how long each phoneme lasts, a prosody branch predicts pitch and energy, and a decoder plus an iSTFT generator — inverse short-time Fourier transform, the step that turns per-frame spectra back into samples — write the waveform. Pocket TTS is a different lineage. A small language model reads the text as tokens and predicts, frame by frame, a compressed picture of the sound; a codec decoder turns those frames into samples. It has no phoneme step, and it speaks in a voice it takes from a few seconds of audio. caps reports the rate each model emits; all four emit 24 kHz.

Run it with a TTS GGUF — the phoneme families want their two front-end packs beside the file, a Pocket file stands alone:

daslang.exe -jit tutorials/dasLLAMA/16_text_to_speech.das -- kitten-nano.gguf hello.wav
daslang.exe -jit ... -- kokoro-82m.gguf hello.wav --voice af_heart
daslang.exe -jit ... -- kitten-nano.gguf hello.wav --f32
daslang.exe -jit ... -- pocket-tts-en-q8.gguf hello.wav --clone me.wav

Use -jit. Speech synthesis runs the same tuned kernels the language models run, and the interpreter is far too slow for them.

8.7.17.1. One loader, the packs beside the model

load_tts_model reads the family from the GGUF’s own architecture metadata, so no family name appears in the call. Two more files travel with the model: tts_g2p.bin and tts_postag.bin, and the loader reads them from the model’s directory. A file that is not a TTS family panics and names the architecture it found. g2p_pack_path names the phoneme pack the loader takes from a directory: tts_g2p.bin when it is there, else the American-only tts_g2p_en_us.bin - the smaller pack a browser build ships, which reads no British voice.

var m <- load_tts_model("kitten-nano.gguf")   // tts_g2p.bin + tts_postag.bin sit beside it
print("phoneme pack: {base_name(g2p_pack_path(dir_name("kitten-nano.gguf")))}\n")

A Pocket file reads text, so no pack sits beside it. tts_needs_packs answers that from the file’s own architecture metadata, before any load, so a program can check the file set it is about to serve.

if (tts_needs_packs(path)) {
    print("wants tts_g2p.bin + tts_postag.bin beside it\n")
}

8.7.17.2. caps(): ask, don’t assume

caps answers what the loaded model serves: the voice names it accepts, the language each one speaks, the rate of the PCM it hands back, and whether it can clone a voice from a recording. Ask for a voice or a language caps does not carry and the call panics — the request is never quietly swapped for something the model does have.

voices can come back empty. A model file may carry packs whose languages this front end does not phonemize — Kokoro ships fifty-four in nine languages, and caps lists the twenty-eight it can drive — so a build that phonemizes none of them offers none, and there is no first voice to fall back on. Check before you index.

let c <- caps(m)
print("{length(c.voices)} voices, {c.sample_rate} Hz, clones a voice: {c.cloning}\n")
if (empty(c.voices)) {
    print("this model has no voice the front end can drive\n")
    return
}
let voice = c.voices[0]

8.7.17.3. Speak it

synthesize takes the text and hands back TtsAudio: mono f32 PCM, the sample rate, and the timings. write_wav_pcm16 writes that PCM as a 16-bit RIFF/WAVE file, which is what a player expects.

let audio <- synthesize(m, "daslang speaks.", voice)
write_wav_pcm16("hello.wav", audio.pcm, audio.sample_rate)

8.7.17.4. What the model receives

The two packs beside the GGUF are the text front end, and they run before any model weight is touched. tts_postag.bin carries a tokenizer and a part-of-speech tagger; tts_g2p.bin carries the lexicons that turn words into phonemes. Three passes run in order: normalization rewrites numbers, money, dates, times and abbreviations into the words a reader says, the tagger labels each token, and the grapheme-to-phoneme pass turns the labelled tokens into the 45 US phoneme symbols the model was trained on. The tag is what settles a heteronym — read as a verb and read as a past tense are the same letters and different sounds.

synthesize runs all three for you. Two facade calls run them alone, so you can see what the model is asked to say. tts_normalize is the first pass, and it needs no model — the normalizer is rules, not a pack.

print("{tts_normalize("Dr. Chen read 3.5 pages and paid $12.")}\n")
// output: Doctor Chen read three point five pages and paid twelve dollars.

tts_phonemize is the last pass, and it takes one already-normalized sentence — the same piece synthesize hands the model. It answers in the front end’s own inventory, before any family rewrites those symbols into the ones it was trained on, so two models with different symbol tables give you the same string here.

let spoken = tts_normalize("Dr. Chen read 3.5 pages and paid $12.")
print("{tts_phonemize(m, spoken)}\n")
// output: dˈɑktəɹ ʧˈɛn ɹˈɛd θɹˈi pˈYnt fˈIv pˈAʤᵻz ænd pˈAd twˈɛlv dˈɑləɹz.

read comes out ɹˈɛd, the past tense, because the tagger called it one.

That two-argument form reads American English. A synthesis, though, phonemizes in the dialect the voice was trained on — Kokoro’s bf_emma reads en-gb where af_heart reads en-us — so a document about what a particular voice will say has to ask in that voice’s language, or it describes a different sound. caps carries one language per voice in voice_langs, beside voices; tts_voice_lang answers for a family alias too, and panics on a voice the model cannot drive with the message a synthesis would give.

let lang = tts_voice_lang(m, voice)          // "en-us" | "en-gb"
print("{voice} reads {lang}\n")
print("{tts_phonemize(m, spoken, lang)}\n")  // the string that voice is asked to say

A Pocket model reads text, not phonemes. tts_has_phonemes answers false for it, and tts_phonemize panics rather than invent a string. The English normalizer still runs in front of the English file; the other languages read their text as it is.

if (tts_has_phonemes(m)) {
    print("{tts_phonemize(m, spoken, lang)}\n")
} else {
    print("{spoken}\n")   // the text itself is what the model reads
}

The front end is where synthesize spends its first microseconds, and TtsTimings counts them separately from the model stages.

8.7.17.5. Where the time went

TtsTimings is microseconds of wall clock per stage, model loading excluded: the front end, then bert — a phoneme-level language model that gives each phoneme the context of its neighbours — the text encoder, the durations, the prosody branch, the decoder, the source and the generator. A Pocket synthesis fills four slots of its own — the prompt, the backbone, the head and the codec — and leaves the others at zero. rtf is the real-time factor, seconds of work per second of audio produced, so a number below 1 is faster than real time. timings_line puts the stages a family ran on one line.

print("{timings_line(audio.timings, audio.sample_rate)}\n")
print("rtf {rtf(audio.timings, audio.sample_rate)}\n")

8.7.17.6. One sentence at a time

synthesize_stream runs the same pieces and hands each one to the block as soon as it exists, in order. A player can start on the first sentence while the second is still being made, so the wait a listener feels is the first chunk, not the whole text. Both forms walk the same chunk list, and each chunk draws its source noise from its own index, so the streamed audio and the buffered audio are the same samples.

synthesize_stream(m, "First sentence. Second sentence.", voice, 1.0) $(chunk) {
    print("{length(chunk.pcm)} samples at {chunk.sample_rate} Hz\n")
}

8.7.17.7. The chunker

One synthesis takes one sentence-sized piece, and tts_chunks is the splitter synthesize runs over the normalized text. It cuts at sentence ends and splits a sentence past TTS_CHUNK_CHARS codepoints on a word boundary. Given the model, it follows that family’s own driver: Kitten’s appends a comma to a piece the split left bare, so the model hears a prosodic close; Kokoro’s sends the text as it is, because its voices read an added mark out loud as a breath. The cap counts codepoints, not bytes — an em dash costs three bytes and one character. A period inside Dr. or 3.5 is not a sentence end.

for (piece in tts_chunks(m, "Dr. Smith read it. It ran to 3.5 pages.")) {
    print("{piece}\n")
}
// -> "Dr. Smith read it."
// -> "It ran to 3.5 pages."

8.7.17.8. Speed and voice

speed scales the predicted durations, so the same words come out in less time at the same pitch. The voice is a name from caps().voices; each one is a style vector the model was trained with, and switching costs nothing but the lookup. A KittenTTS voice carries its own speed prior on top of the number you pass, so the same speed on two voices is not the same tempo — hold the voice fixed when you want to read the knob.

caps().speed says whether the model honours a speed at all. Pocket TTS has nothing that scales a duration, so it reads false there, and a speed other than 1.0 panics.

if (c.speed) {
    let quick <- synthesize(m, "daslang speaks.", voice, 1.25)
}
let other <- synthesize(m, "daslang speaks.", c.voices[1])

8.7.17.9. Cloning a voice

caps().cloning says whether the model takes a voice from a recording. A Pocket TTS file with its codec encoder does: a few seconds of one speaker, mono, at the model’s own rate, become a voice in the roster. load_audio_mono decodes a wav, flac, mp3 or ogg file to that rate, and tts_register_voice adds the samples under the name you give. From then on the name works like any bundled voice. A clip longer than a minute is refused; a Pocket file converted without the encoder (a one-voice file for a page) reports cloning false and refuses by name; a phoneme model panics here: it has no voice to take.

if (c.cloning) {
    let clip <- load_audio_mono("me.wav", c.sample_rate)
    tts_register_voice(m, "me", clip, c.sample_rate)
    let mine <- synthesize(m, "daslang speaks in my voice.", "me")
}

8.7.17.10. Giving the scratch back

A synthesis leaves its activation rows inside the model, sized to the largest one so far, so the next one pays no allocation. On a phone’s browser tab that idle memory is what runs out first. tts_release_scratch gives the rows back, and the next synthesis grows them again for a few milliseconds. Call it when nothing is queued: the browser examples’ speech threads call it after half a second with no request waiting.

tts_release_scratch(m)
let again <- synthesize(m, "It speaks as before.", voice)

The rows are sized by the longest chunk a synthesis speaks, so the peak a say holds is the chunk cap’s. A Kitten or Kokoro model speaks a sentence of up to TTS_CHUNK_CHARS codepoints in one piece; tts_set_chunk_chars lowers that, and the same text speaks in more, shorter pieces at a lower peak. Each piece is its own synthesis, so a phrase cut in two is read as two phrases.

tts_set_chunk_chars(m, 120)
let low <- synthesize(m, "The same sentence, spoken in shorter pieces at a lower peak.", voice)

8.7.17.11. Prepared voice prompts

tts_prepare_voice_prompt encodes a reference into a TtsVoicePrompt. This step needs a Pocket model with its codec encoder. The value can be serialized with daslib/json_boost and shipped separately from the model weights. tts_voice_prompt_valid checks the format, language, sample rate, codec geometry and bounded latent frames against the receiving model.

tts_register_voice_prompt copies a prepared prompt into a named voice. It also works with a compatible encoder-free Pocket file. An empty name or invalid prompt returns false and leaves the previous voice intact. The caller owns its prompt; the model builds conditioning state when the voice is first synthesized. Matching geometry does not make prompts portable across unrelated model revisions.

The tutorial’s --clone path also demonstrates a JSON round trip and writes speech from the restored prompt:

let clip <- load_audio_mono("me.wav", c.sample_rate)
var inscope prompt <- tts_prepare_voice_prompt(m, clip, c.sample_rate)
var inscope restored = TtsVoicePrompt()
if (sscan_json(sprint_json(prompt, false), restored)
        && tts_voice_prompt_valid(m, restored)) {
    if (tts_register_voice_prompt(m, "prepared", restored)) {
        let speech <- synthesize(m, "daslang speaks.", "prepared")
        write_wav_pcm16("prepared.wav", speech.pcm, speech.sample_rate)
    }
}

8.7.17.12. The weight lanes

The decoder and generator matrix multiplies are served from one of two prepared images beside the GGUF. The q8 lane holds those weights as Q8_0 quants and is what a load serves by default. The f32 lane holds the file’s own planes; it is the reference the parity tests hold the q8 lane against. A published Pocket file already holds Q8_0 weights, so its q8 lane reads them as they are and its f32 lane dequantizes them. The small Pocket files hold Q4_K planes for the backbone and the codec transformers too: an unpinned load serves those planes as they are through the engine’s K-quant kernels, a third lane, while a pin to q8 or f32 requantizes or dequantizes them at load.

tts_serves_q8 answers which lane the next load takes. set_tts_q8 pins it, and reset_tts_q8 returns to the default. The pin is process-wide state, so it must not outlive the load it was set for: a stale pin changes the lane of the next model this process loads. Pin through defer and every path out resets it, a panic included.

defer() {
    reset_tts_q8()
}
set_tts_q8(false)                 // the f32 reference lane, for this load only
print("next load serves q8: {tts_serves_q8()}\n")
var reference <- load_tts_model("kitten-nano.gguf")