Cornserve: Rethinking Distributed Inference When Every Request Takes a Different Path
I wasn’t able to fetch the full paper, so I’ll write based on the abstract and technical context. I’ll avoid attributing specific numbers to the paper that I can’t verify.
The Problem with Serving Models That Can Do Everything
Most inference serving infrastructure was built with a tacit assumption: a model has a fixed input type and a fixed output type. Text in, text out. Image in, caption out. That assumption is quietly breaking down.
Any-to-Any multimodal models—systems like the one Cornserve targets—accept arbitrary combinations of text, images, video, and audio as input and can generate any of those modalities as output. A single deployed model might handle a text-only Q&A request, a video captioning job, an audio transcription, and a request that takes an image plus text and returns a video clip—all concurrently. Serving infrastructure that treats all of this as “just tokens through a transformer” will leave significant efficiency and latency on the table.
Why the Computation Graph Is the Problem
The fundamental issue Cornserve addresses is that multimodal requests don’t share a uniform execution path. A request that receives only text and returns only text might skip the vision encoder, the audio encoder, and the video decoder entirely. A request that takes video as input and returns audio does the opposite. The model’s computation graph is not a single pipeline—it’s a DAG with branches that different requests activate selectively.
This creates two intertwined problems for a serving system:
Variable path traversal. Batching—the primary tool for GPU utilization in LLM serving—works best when requests execute the same operations. When requests diverge across modality branches, naive batching either wastes compute (padding absent modalities) or fails to batch effectively at all.
Heterogeneous component scaling. A vision encoder, a large language model backbone, and an audio decoder have radically different arithmetic intensity, memory bandwidth requirements, and latency profiles. The LLM backbone is memory-bandwidth-bound and benefits enormously from large batch sizes and KV-cache management. A video encoder is compute-intensive and may benefit from different parallelism strategies. An audio decoder might be relatively lightweight but latency-sensitive. No single scaling strategy fits all of these simultaneously.
These aren’t new observations in isolation—video pipelines and audio systems have always had heterogeneous stages—but Any-to-Any models collapse them into a single model artifact that must be served as a coherent unit.
What Cornserve Does Differently
Cornserve is designed around the insight that Any-to-Any models are better understood as distributed systems problems rather than single-model inference problems. Rather than deploying the whole model as a monolith and hoping a unified scheduler handles heterogeneity gracefully, Cornserve treats each modality component as a separately scalable service.
This maps naturally onto what engineers already do with microservices: identify components with different resource profiles, deploy and scale them independently, and route requests through the appropriate subgraph. The challenge Cornserve solves is doing this automatically for arbitrary model architectures, without requiring the developer to manually decompose the model or write routing logic.
The system’s routing layer must reason about which components a given request will activate, then dispatch work to the right component replicas while maintaining the coordination needed to reassemble results. Critically, it has to do this without introducing prohibitive overhead—the dispatcher can’t be a bottleneck when individual component latencies may already be tight.
Scaling decisions also become dynamic: if a workload shifts toward image-heavy requests, the vision encoder pool should grow without requiring the LLM backbone to be re-provisioned. This kind of per-component autoscaling is only possible if the serving system has a clean abstraction for modality components as independent units.
Implications for Inference Infrastructure
Cornserve represents a broader architectural shift worth tracking. Current production inference stacks—vLLM, TensorRT-LLM, SGLang—are largely optimized for the LLM-centric case, with multimodal support grafted on. As Any-to-Any models move from research artifacts toward deployment, serving systems will need to either extend these tools or adopt component-decomposed approaches like Cornserve’s.
For developers building on top of hosted inference APIs, this matters less immediately—the complexity is absorbed by the provider. But for teams running their own inference clusters with open-weight multimodal models, the gap between “this model works in a notebook” and “this model serves production traffic efficiently” is going to widen unless the tooling catches up.
Watch for whether Cornserve’s component-decomposition approach influences how model repositories package Any-to-Any models—if component boundaries become a first-class artifact in model configs (analogous to how tensor parallelism hints are sometimes included today), that would signal the paradigm taking hold.