Framework
TensorRT LLM
TensorRT LLM is an open-source AI framework from NVIDIA, launched in 2023, made for developers and businesses.
A library for optimizing and serving LLM inference on NVIDIA GPUs.
Main jobs: Model serving. It is open source and free to use. You can use it on your own server, through an API and in the terminal.
Markdown version of https://godooo.ai/en/framework/tensorrt-llm. Every page on this site has one: add .md to its address.
At a glance
- Type
- Framework
- Made by
- NVIDIA
- Launched
- 2023
- Galaxy
- Models & APIs
- Open source
- Yes
- Source code
- github.com/NVIDIA/TensorRT-LLM
- GitHub stars
- 14,709
- How you run it
- Self-host · API · Terminal
- Made for
- Developers · Businesses
- Website
- nvidia.github.io
- Also known as
- TensorRT-LLM
What it can do
Model servingmain jobunconfirmed
“TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework”— github.com, 2026-09-24
LLM APIsupportedunconfirmed
“LLM API Introduction API Reference LLM AsyncLLM”— nvidia.github.io, 2026-09-24
unconfirmedproposed by a machine, awaiting calibration
Alternatives
- vLLMOpen source, freeAlso does: Model serving, LLM API; runs offline
- SGLangOpen source, freeAlso does: Model serving, LLM API; runs offline
- LocalAIOpen source, freeAlso does: LLM API, Model serving; runs offline
- LMDeployOpen source, freeAlso does: LLM API, Model serving
- exoOpen source, freeAlso does: Model serving, LLM API; runs offline
- MLC LLMOpen source, freeAlso does: Model serving, LLM API; runs offline
- falPaidAlso does: Model serving
- ReplicatePaidAlso does: LLM API, Model serving
- Together AIPaidAlso does: LLM API, Model serving
- ModalPaidAlso does: Model serving
- Hugging FaceFree plan + paid upgradesAlso does: LLM API, Model serving; runs offline
- OllamaOpen source, freeAlso does: LLM API, Model serving; runs offline
Connections
- Built with
- CUDA
Questions
What is TensorRT LLM?
TensorRT LLM is an open-source AI framework from NVIDIA, launched in 2023, made for developers and businesses. A library for optimizing and serving LLM inference on NVIDIA GPUs.
Is TensorRT LLM free?
Yes. TensorRT LLM is open source and free to use.
Is TensorRT LLM open source?
Yes. The source code is at https://github.com/NVIDIA/TensorRT-LLM.
Can TensorRT LLM run locally?
TensorRT LLM can be self-hosted on your own server.
What can TensorRT LLM do?
Main jobs: Model serving. It can also do: LLM API.
Who makes TensorRT LLM?
TensorRT LLM is made by NVIDIA.
What are open-source alternatives to TensorRT LLM?
Open-source ones: vLLM, SGLang, LocalAI, LMDeploy, exo, MLC LLM and Ollama. Others: fal, Replicate, Together AI, Modal and Hugging Face.
Sources
- github.com/NVIDIA/TensorRT-LLMchecked 2026-09-24
- nvidia.github.io/TensorRT-LLM/checked 2026-09-24