Streaming playback: why a transcript can arrive without sound

Follow the output all the way to the speaker

A server transcript and a playable response are different events. PocketCook parses inlineData, decodes Base64, validates the PCM format and queues bytes for AudioTrack. The transcript path can succeed while any later audio stage is stalled. Diagnose the stages separately instead of assuming every silent answer is an API-key problem.

The baseline accepts audio/pcm or audio/pcm;rate=24000, even byte lengths and at most 256,000 decoded bytes per part. Unsupported data ends the session with a safe message. The exact accepted strings are the sample's contract; changing provider formats requires a deliberate parser update and tests.

Open the diagram at full size · Mermaid source

Two timelines run concurrently

The network produces chunks. The device consumes frames. Network turnComplete means a model turn ended; the hardware may still have buffered samples to play. In LiveConnection.kt, that event does not forcibly set speaking to false. The audio adapter observes playbackHeadPosition and pending output separately.

Copy code// AndroidPcmAudio.kt — configuration excerpt
AudioTrack.Builder()
    .setAudioAttributes(attributes)
    .setAudioFormat(
        AudioFormat.Builder()
            .setSampleRate(24000)
            .setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
            .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
            .build()
    )
    .setTransferMode(AudioTrack.MODE_STREAM)
    .setBufferSizeInBytes(maxOf(outputSize, 4800))
    .build()

At 24 kHz mono PCM16, 4,800 bytes represent 100 ms and 2,400 frames. One frame is one sample for this mono stream. Confusing bytes with frames doubles or halves your completion estimate.

The startup stall

The original writer fed one small chunk, then waited for that chunk to drain before reading the next. Some devices need a startup threshold of buffered frames before playback begins. If the first chunk was smaller than that threshold, the writer waited for playback and playback waited for more bytes: neither could progress.

The fix keeps feeding subsequent chunks without a per-chunk drain wait. On API 31+, the sample also requests a one-frame start threshold. The continuous writer is still necessary on older devices and while partial writes occur. The AudioTrack reference explains streaming start thresholds and write behavior.

Copy code// Baseline policy; keep this API guard
if (Build.VERSION.SDK_INT >= 31) {
    track!!.setStartThresholdInFrames(1)
}

A nonblocking write may consume fewer bytes than requested or zero. Advance the offset only by the successful return count; when zero, briefly yield and retry. Negative return values take the failure path. Never assume a single write consumed the entire chunk.

Interruptions must invalidate work already in flight

Flushing a channel is insufficient if the writer already dequeued an old chunk. Each queued output carries an epoch. Interruption increments the epoch, drains the queue and pauses/flushes/restarts the track. The writer checks the epoch inside its write loop, so its remaining old bytes are discarded too.

This protects locally queued audio; it does not label future server packets with turn identities that the protocol did not provide. State that limitation when reasoning about network ordering. Old sessions are guarded separately by connection generation.

What the playback test proves

AudioPlaybackTest.smallChunksKeepFeedingUntilPlaybackAdvances queues twenty 480-byte silent chunks, each 10 ms, and waits for the playback head to reach 4,800 frames. This reproduces the small-first-chunk condition without playing a test tone. It is stronger than checking that play() was called, because the frame position must advance.

It still does not prove a human heard Gemini correctly, that the speaker route was appropriate, or that the microphone did not capture an echo. Those require a separate live-device run. The companion sustained test checks capture chunk size, mute/resume and a fresh engine across two sessions, without sending audio to a server.

A useful debugging order

Check connection/setup, parsed audio parts, PCM validation, queue acceptance, write progress, playback-head progress, and finally route/volume. Use counters and timings rather than logging keys, PCM or conversations. Predict which stages can pass when the UI shows a transcript but the device is silent. The answer is several: only the transcript path is demonstrated by the text.

Course study guide · Hands-on codelab · Pinned Android source