Stop Re-Prefilling: Quantized KV-Cache Handoff for Multi-Agent Edge LLMs
I don’t have access to fetch the paper, so I’ll write the explainer based on the abstract and my knowledge of the relevant techniques.
The Problem: Context Handoff Is a Bottleneck for Multi-Agent Edge AI
As LLM-powered apps shift from single-model pipelines to multi-agent architectures—where a planner delegates subtasks to specialist agents—a deceptively hard systems problem emerges: how do you pass context between agents efficiently? On a cloud server with fast interconnects and abundant VRAM, it barely registers. On an edge device—a phone, a laptop, an embedded system—it becomes a first-class constraint.
The naive solutions are both painful. Re-prefill means the receiving agent re-processes the entire prior context from scratch, burning compute and latency before it can even start its actual task. Full-precision KV transfer avoids redundant computation, but shipping raw float16 or bfloat16 key-value tensors between agents is bandwidth-heavy—on devices where memory bandwidth is the scarcest resource, this is often worse.
QKVShare proposes a third path: quantize the KV cache before handoff, transfer a compact self-describing bundle, and inject it directly into the receiving agent’s attention mechanism.
What the KV Cache Actually Is
To understand why this works, it helps to be precise about the KV cache. During a transformer’s attention computation, each token in the prompt produces a key vector and a value vector at every layer. Caching these lets the model avoid recomputing them for tokens it has already seen—critical for generation speed. The cache for a 2048-token context in a 7B-parameter model with 32 layers can easily exceed several hundred megabytes in float16.
When agent A finishes processing a document and hands off to agent B, B needs that same contextual representation. Without the cache, it re-reads the document. With the raw cache, it gets a large tensor blob with no attached metadata. What QKVShare introduces is a disciplined middle ground.
Three Interlocking Mechanisms
Token-level mixed-precision allocation is the first key idea. Not all tokens in a KV cache are equally important—attention sinks (often the first few tokens), high-salience content, and recent tokens tend to matter more than filler in the middle. QKVShare assigns quantization precision per-token rather than globally, so salient tokens get INT8 while less critical ones drop to INT4. This lets the framework hit aggressive compression ratios without uniformly degrading the representation quality that downstream agents depend on.
CacheCard is the serialization format that makes handoff practical. Rather than a raw tensor dump, a CacheCard is a self-contained bundle: the quantized KV tensors, the per-token precision metadata, the quantization parameters (scales and zero points), and enough model-architecture metadata that a receiving agent can reconstruct the cache without out-of-band coordination. Think of it as a typed, versioned cache snapshot—closer to a structured message than a memory dump. This matters for real deployments where agents might not share a process or even run the same model variant.
HuggingFace-compatible cache injection closes the loop on practicality. The Transformers library has a well-defined Cache abstraction—DynamicCache, StaticCache, and friends—that most open-weight model inference code already uses. QKVShare implements an injection path that deserializes a CacheCard into this interface, meaning the receiving agent’s generation loop sees a populated cache and proceeds as if it had prefilled the context itself, with no changes to the model’s attention code.
What the Numbers Show
The paper evaluates on 150 GSM8K math reasoning problems—a benchmark that stresses multi-step coherence, exactly the kind of context that needs to survive handoff intact. The results confirm the core tradeoff: quantized KV handoff recovers most of the quality of full-precision transfer at substantially lower bandwidth cost. Precise accuracy figures depend on the precision configuration, but the headline claim is that token-level mixed precision outperforms uniform quantization at the same average bit-width, validating the salience-aware allocation strategy.
The paper is candid that the story is “narrower but clearer” than an earlier draft—the evaluation is deliberately focused rather than sweeping. That’s actually a useful signal: the authors are reporting what they can substantiate rather than over-claiming.
Why Edge Deployment Makes This Non-Trivial
On-device LLMs introduce constraints that cloud deployments don’t face. Memory is limited and often non-expandable. Inter-process communication doesn’t have the luxury of NVLink or fast PCIe lanes—on mobile SoCs you’re moving data over the same bus that feeds the display, the camera, and the modem. Quantization that looks like a minor accuracy tradeoff in a benchmark can compound across multi-turn agent interactions where errors accumulate. And unlike a server, you can’t just scale out.
QKVShare’s design reflects these constraints: the CacheCard format is designed for transfer, not just storage; the mixed-precision policy is adaptive rather than fixed; and the HuggingFace injection path means adoption doesn’t require forking model code.
What to Watch
Three open questions will determine how far this approach scales. First, how well does the salience-based precision allocator generalize—the policy presumably learned or tuned on some distribution of prompts may not transfer cleanly to domain-specific agent pipelines. Second, the interaction with model quantization: if the agent models themselves are already quantized (common on-device), stacking KV quantization introduces compounding error that needs careful characterization. Third, multi-hop handoffs—where context passes through three or more agents—have potentially non-linear quality degradation that single-hop benchmarks won’t surface.
For developers building multi-agent pipelines targeting edge hardware, QKVShare is worth tracking closely. The HuggingFace-compatible design means integration overhead is low, and the CacheCard abstraction is the kind of portable primitive that could become a standard interchange format as on-device agent frameworks mature.