From Raw Speed to Composability: TileRT × vLLM Heterogeneous PD Disaggregation

In production deployments today, the prefill and decode phases stress different resources: prefill is dominated by compute, while decode is bound by memory bandwidth. Physically decoupling the two — prefill–decode (PD) disaggregation — keeps these two kinds of work from dragging each other down inside the same execution window.

When it comes to actually deploying this, however, users face a very practical engineering choice. TileRT's low-level decode is fast, but most teams already run a mature serving stack built around vLLM: central scheduling, prefix caching, and standard APIs that are already wired into their products. Rebuilding the upper serving stack just to bring in TileRT's low-latency decode usually means a steep migration cost.

The TileRT × vLLM PD disaggregation solution exists precisely to combine the two — without breaking the native vLLM ecosystem.

In short: a lightweight external router takes in traffic, vLLM handles scheduling and computation for the prefill phase, and TileRT picks up decode on the back end. Each engine does what it does best, cooperating through official, open interfaces. Users no longer have to choose between “the native ecosystem” and “extreme speed.”

Zero Modification: A Plugin on the Official Extension Point

To leave vLLM's codebase clean and future upgrades painless, this solution modifies none of vLLM's core source code. It is built entirely on the official, stable extension point vLLM has provided since the V1 architecture: KVConnectorBase_V1.

We implemented a custom TileRTConnector (acting in the kv_producer role) in tilert_serve/pd_vllm/prefill_connector.py. At launch time, injecting the following --kv-transfer-config into the standard vLLM serve command is all it takes to load the module dynamically:

--kv-transfer-config '{
    "kv_connector": "TileRTConnector",
    "kv_connector_module_path": "tilert_serve.pd_vllm.prefill_connector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
        "tilert_host": "YOUR_TILERT_HOST",
        "tilert_ctrl_port": 5556,
        "tilert_model": "glm5",
        "tilert_max_seq_len": 202752
    }
}'

From the perspective of vLLM's central scheduler, the TileRT integration layer is simply another transfer plugin conforming to the standard lifecycle contract — plug and play.

One more required setting: because the decode side runs multi-token prediction (MTP) speculative decoding from the very first step, the prefill side must also carry the matching speculative config (--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}') — the draft-layer KV is populated by vLLM during prefill, and decode-side speculation resumes exactly from there.

State Transfer: An Async Mooncake RDMA Data Plane, Zero Forward Blocking

Once a prefill node finishes computing, the request's attention state — compressed KV, sparse-attention index caches, and a small amount of metadata — has to move to the TileRT decode node behind it. In ultra-low-latency scenarios, cross-engine network synchronization and data movement can easily introduce execution overhead and latency jitter.

To drive communication cost to a minimum, the system layers two designs across the data plane and the execution timeline:

• RDMA one-sided writes, never touching host memory: the data plane supports both Mooncake and NIXL as transfer engines. After computation on the vLLM side, state is written directly into pre-registered GPU memory on the TileRT decode node via RDMA one-sided WRITE — no intermediate serialization, no staging through host memory. The handoff protocol itself is decoupled from the underlying transfer engine, whose only job is to move bytes.
• Zero blocking of the forward window: inside vLLM's normal forward execution window, we only do a fast extraction of the state into a staging buffer — completed before the corresponding cache blocks can be recycled. The data is then immediately handed to an independent background async sender that triggers the actual network transfer; the forward pass itself never stalls waiting on the network. A request bound for TileRT therefore never slows down other native requests in the same batch, eliminating the risk of queuing behind the network.

State Restoration: Injection into a Live Engine

When the state arrives on the TileRT side, it is first converted into TileRT's native memory layout, then injected directly into a live, running engine instance. Decode starts immediately after injection, with multi-token prediction (MTP) active from the very first step — no engine cold start, no scheduling bubbles.

Routing Isolation: Multi-Pool Coexistence and an Adaptive Claim Mechanism

In real cluster deployments, the native vLLM decode pool and the TileRT decode pool need to coexist safely within the same cluster — even within the same forward batch.

Rather than modifying vLLM's native central scheduler, we achieve this with an adaptive claim-filter mechanism that isolates request ownership inside vLLM:

• External routing decisions: routing happens outside vLLM, in an externally deployed router. On receiving a request, the router forwards it to vLLM for prefill but forces max_tokens to 1 (vLLM performs the prefill and emits exactly the first token). It also attaches a special pass-through mark to the request: kv_transfer_params = {"tilert_host": ..., "tilert_ctrl_port": ...}.
• Precise claim filtering: inside vLLM, we use the official MultiConnector to place the native transfer plugin side by side with our TileRTConnector. Our connector provides a static filter, _claim: it claims only requests carrying the tilert_host mark; anything without the mark gets a strict no-op (silent non-interference).

With this design the two marks are mutually exclusive and their namespaces disjoint: marked requests go to TileRT, unmarked ones go to the native pool, and the two can run mixed in the same forward batch without interfering. Meanwhile, on the decode side, the TileRT pool has its own gated-dispatch mechanism for back-pressure, fully isolated from the vLLM pool's unbounded-queue scheduling — so a backlog on the backend can never degrade latency for the rest.

Wrapping Up

We have verified the full pipeline end to end on real models: after state injection, the decode side fully preserves TileRT's native single-stream generation speed, MTP takes effect from the first step, and the cross-engine handoff introduces no perceptible performance loss. Models supported in the current release are GLM-5/5.1 and DeepSeek-V3.2, with more on the way.

This TileRT × vLLM collaboration offers a reference blueprint: concurrency optimization, prefix caching, and standard APIs in the prefill phase go to vLLM; squeezing decode latency to the limit goes to TileRT. The two sides divide the work through open, well-defined interfaces — keeping the mainstream ecosystem while unlocking the full performance of the underlying hardware.

Get in Touch

The TileRT team will keep exploring the foundations and performance limits of high-performance AI systems, and we warmly welcome deep collaborations with model teams and AI application teams.

If you are interested in GPU architecture, compiler optimization, distributed systems, or large-scale inference, we would also love to have you on board.

• GitHub repo: github.com/tile-ai/TileRT
• Contact / join us: contact@tilert.ai