How prompt layout determines what your KV cache can reuse
Systems such as vLLM and SGLang can reuse cached K/V tensors for the longest contiguous matching prefix of each new prompt. A change early in the template invalidates everything after it, so TTFT gains depend on layout as much as on whether two prompts are “almost the same” in aggregate. Below are a few results from one RAG-QA sweep: Natural Questions-style prompts (~340 tokens on average); four-part structure (system instruction, user query, retrieved chunks, output instruction); TinyLlama-1.1B on one A100-40GB via vLLM.
I compared four mutations—typo and synonym_substitution on the query (early), formatting on the output block (already at the tail), and chunk_reorder inside the retrieval section (bulk of the tokens)—against two layouts: the original ordering and stable_first, which moves whichever block is mutated to the very end. One thousand paired examples per cell; cache flushed between comparisons.
Shared prefix grows when volatile text moves to the tail
With stable_first the shared-prefix fraction tracks how large the mutated region is relative to the full prompt—you cannot reuse more than the genuinely unchanged portion.
- typo / synonym_substitution (~10 volatile tokens early): shared prefix about 0.06 → ~0.98.
- formatting (change already at the tail): ~0.95 → ~0.95 (layout is effectively a no-op).
- chunk_reorder (most of the prompt is the chunk block): ~0.15 → ~0.19; little headroom because the volatile region dominates.
TTFT gain scales ~linearly with cached tokens (~17 μs per token)
Pooling across conditions, TTFT savings track how many prefix tokens are served from cache. The linear fit lands near seventeen microseconds per cached token with intercept roughly zero (−0.3 ms, within noise). A first-order division from the setup yields similar numbers: roughly 2.2 GFLOPs per prefilled token and effective BF16 throughput on the order of 128 TFLOPS on an A100 (about forty percent utilization of nominal peak here) yields ~17.2 μs per token—versus ≈17.0 μs from the slope—consistent with substitution of prefill with a comparatively cheap KV lookup rather than unexplained overhead.
Layout captures most upside when volatility is localized
Combining the two plots: recoverable TTFT is bounded by how much of the prompt can stay verbatim at the shared prefix. For small early mutations (typo, synonym substitution), stable_first lifts outcomes from near the measurement floor (~0.05 ms TTFT gain) toward the approximate full-reuse ceiling (~5.5 ms here) without touching the model. For formatting, the edited block already sits at the tail, so the layout pivot is redundant. For chunk reorder, most tokens move, so there is almost no stable prefix left to amortize regardless of ordering.
More workloads, mutations, and analysis scripts live in github.com/fjavid/prefix_caching (including project_description.md for the broader roadmap).