TileRT
Ultra-Low-Latency LLM Inference
A revolutionary tile-level runtime engine that unlocks inference speed for state-of-the-art AI models.
$ pip install tilert
Ultra-Low-Latency LLM Inference
A revolutionary tile-level runtime engine that unlocks inference speed for state-of-the-art AI models.
$ pip install tilertDesigned from the ground up for latency-critical LLM serving, and powered by advanced compiler techniques.
Prioritizes responsiveness over throughput. Achieve millisecond-level time per output token (TPOT) for models with hundreds of billions of parameters.
Decomposes LLM models into fine-grained tile-level tasks and dynamically reschedules computation, I/O, and communication across multiple devices.
Leverages advanced compiler techniques to automatically minimize idle time and maximize hardware utilization through highly overlapped execution across GPUs.
Preserves full model quality and accuracy without any lossy optimizations such as quantization or distillation.
Built for multi-device deployment, with dynamic scheduling that overlaps computation and communication.
TileRT is designed for empowering high-value AI scenarios, such as agents, vibe coding, trading, and real-time decision-making.
TileRT demonstrates substantial speedups in token generation over baselines.
Get TileRT up and running in minutes with Docker and pip.
# Path to the workspace you want to mount WORKSPACE_PATH="xxx" $ docker run --gpus all -it -v $WORKSPACE_PATH:/workspace/ tileai/tilert:v0.1.0
Note: For the most reliable experience, we strongly recommend using the provided Docker image. See the GitHub repository for full documentation.
TileRT is part of a growing ecosystem of compiler and runtime tools for AI workloads.