# TensorRT LLM

Framework · https://godooo.ai/en/framework/tensorrt-llm

> TensorRT LLM is an open-source AI framework from NVIDIA, launched in 2023, made for developers and businesses.

A library for optimizing and serving LLM inference on NVIDIA GPUs.

Main jobs: Model serving. It is open source and free to use. You can use it on your own server, through an API and in the terminal.

## At a glance

- **Type:** [Framework](https://godooo.ai/en/browse#framework)
- **Made by:** [NVIDIA](https://godooo.ai/en/org/nvidia)
- **Launched:** 2023
- **Galaxy:** [Models & APIs](https://godooo.ai/en/galaxy/builders)
- **Open source:** Yes
- **Source code:** [github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)
- **GitHub stars:** 14,709
- **How you run it:** Self-host · API · Terminal
- **Made for:** Developers · Businesses
- **Website:** [nvidia.github.io](https://nvidia.github.io/TensorRT-LLM)
- **Also known as:** TensorRT-LLM

## What it can do

- **[Model serving](https://godooo.ai/en/capability/model-serving)** — main job, _proposed by a machine, awaiting calibration_. “TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework” (read on [github.com](https://github.com/NVIDIA/TensorRT-LLM), 2026-09-24)
- **[LLM API](https://godooo.ai/en/capability/llm-api)** — supported, _proposed by a machine, awaiting calibration_. “LLM API Introduction API Reference LLM AsyncLLM” (read on [nvidia.github.io](https://nvidia.github.io/TensorRT-LLM/), 2026-09-24)

## Alternatives

- **[vLLM](https://godooo.ai/en/framework/vllm)** (Open source, free): Also does: Model serving, LLM API; runs offline
- **[SGLang](https://godooo.ai/en/framework/sglang)** (Open source, free): Also does: Model serving, LLM API; runs offline
- **[LocalAI](https://godooo.ai/en/framework/localai)** (Open source, free): Also does: LLM API, Model serving; runs offline
- **[LMDeploy](https://godooo.ai/en/framework/lmdeploy)** (Open source, free): Also does: LLM API, Model serving
- **[exo](https://godooo.ai/en/framework/exo)** (Open source, free): Also does: Model serving, LLM API; runs offline
- **[MLC LLM](https://godooo.ai/en/framework/mlc-llm)** (Open source, free): Also does: Model serving, LLM API; runs offline
- **[fal](https://godooo.ai/en/product/fal)** (Paid): Also does: Model serving
- **[Replicate](https://godooo.ai/en/product/replicate)** (Paid): Also does: LLM API, Model serving
- **[Together AI](https://godooo.ai/en/product/together)** (Paid): Also does: LLM API, Model serving
- **[Modal](https://godooo.ai/en/product/modal)** (Paid): Also does: Model serving
- **[Hugging Face](https://godooo.ai/en/product/huggingface)** (Free plan + paid upgrades): Also does: LLM API, Model serving; runs offline
- **[Ollama](https://godooo.ai/en/product/ollama)** (Open source, free): Also does: LLM API, Model serving; runs offline

## Connections

- **Built with:** [CUDA](https://godooo.ai/en/framework/cuda)

## Questions

### What is TensorRT LLM?

TensorRT LLM is an open-source AI framework from NVIDIA, launched in 2023, made for developers and businesses. A library for optimizing and serving LLM inference on NVIDIA GPUs.

### Is TensorRT LLM free?

Yes. TensorRT LLM is open source and free to use.

### Is TensorRT LLM open source?

Yes. The source code is at https://github.com/NVIDIA/TensorRT-LLM.

### Can TensorRT LLM run locally?

TensorRT LLM can be self-hosted on your own server.

### What can TensorRT LLM do?

Main jobs: Model serving. It can also do: LLM API.

### Who makes TensorRT LLM?

TensorRT LLM is made by NVIDIA.

### What are open-source alternatives to TensorRT LLM?

Open-source ones: vLLM, SGLang, LocalAI, LMDeploy, exo, MLC LLM and Ollama. Others: fal, Replicate, Together AI, Modal and Hugging Face.

## Sources

- [github.com](https://github.com/NVIDIA/TensorRT-LLM) — checked 2026-09-24
- [nvidia.github.io](https://nvidia.github.io/TensorRT-LLM/) — checked 2026-09-24

---

Last updated 2026-09-24 · sources checked 2026-09-24 · Markdown version of https://godooo.ai/en/framework/tensorrt-llm. Every page on this site has one: add .md to its address.