LLM Inference Engine Showdown: vLLM vs Ollama vs TGI

Last week I needed to deploy a custom model for a client project — not through an API provider, but self-hosted, on their infrastructure, with full control over the data. The kind of setup where you actually have to decide how the model runs, not just which model to call.

I spent two days benchmarking vLLM, Ollama, and Hugging Face TGI on the same hardware, reading through architecture docs, and running load tests. What I found was that the "which engine should I use" question has a surprisingly clear answer once you understand what each one is actually built to do.

This guide is the comparison I wish I had before I started. No fluff, real numbers, and a decision framework you can use today.

The Short Version

If you need to ship a production API that handles concurrent users, use vLLM. If you want to run models locally for development or prototyping, use Ollama. If you already have TGI in production, keep it running but plan your migration — Hugging Face themselves put TGI into maintenance mode in December 2025 and now recommend vLLM or SGLang for new deployments.

That covers 90% of decisions. The rest of this article explains why.

What These Engines Actually Do Differently

All three load model weights onto a GPU and generate tokens. The difference is in how they manage memory, handle multiple requests, and scale beyond a single machine.

vLLM was built at UC Berkeley as a research project focused on one thing: serving more concurrent requests per GPU. Its core innovation is PagedAttention — instead of allocating one big contiguous block of GPU memory per request, it splits the KV cache into small fixed-size blocks and allocates them on demand, like virtual memory in an operating system. This eliminates memory fragmentation and lets you fit significantly more concurrent requests into the same VRAM. On top of that, vLLM uses continuous batching, where new requests are inserted into a running batch at every generation step rather than waiting for the current batch to finish.

Ollama is built on top of llama.cpp and designed for a completely different job: making it dead simple to run models locally. You install it, run ollama pull llama3.2, and you have a working model with an OpenAI-compatible API on localhost. It manages model downloads, versioning, and basic GPU scheduling. When a model fits on one GPU, Ollama keeps it there. When it does not, it spreads layers across multiple GPUs — but this is layer offloading, not tensor parallelism. There is no PagedAttention, no continuous batching, and no multi-node support.

TGI sits between the two in design intent. Hugging Face built it as a production inference server with a Rust-based HTTP router handling queuing and batching, and a Python/gRPC model server running the actual inference. It implements continuous batching and uses vLLM's PagedAttention CUDA kernels for memory management. TGI has strong observability out of the box — Prometheus metrics and OpenTelemetry tracing are built in. The catch is that as of December 11, 2025, TGI entered maintenance mode. Only minor bug fixes and documentation PRs are being accepted. For new Inference Endpoints, Hugging Face explicitly recommends vLLM or SGLang.

The Numbers That Actually Matter

Feature tables are easy to find. What is harder to find are real benchmark comparisons on identical hardware. Here is what the data shows.

Throughput Under Concurrency

This is where the engines diverge dramatically. Red Hat published benchmarks comparing vLLM and Ollama on the same hardware, and the results are not close: vLLM peaked at 793 tokens per second while Ollama managed 41 tokens per second. The P99 time-to-first-token was 80ms for vLLM versus 673ms for Ollama.

Independent benchmarks from Clore.ai tell a similar story. Running Llama 3.1 8B on an RTX 4090 with a single user, Ollama produced 65 tokens per second, TGI produced 110, and vLLM produced 140. Scale that to 10 concurrent users and the gap widens: Ollama dropped to roughly 150 tokens per second total (processing requests sequentially), while vLLM hit 800 and TGI reached 500.

A 2025 arXiv study comparing vLLM and TGI specifically found up to 24x higher throughput for vLLM under high concurrency, though TGI showed lower tail latencies in single-user interactive scenarios.

Concurrency Behavior

This is where architecture decisions become visible in production. vLLM's throughput scales almost linearly with concurrency until GPU saturation. Time-to-first-token stays low and inter-token latency rises gradually but remains stable.

Ollama's throughput plateaus quickly as concurrency increases. When you tune the OLLAMA_NUM_PARALLEL setting higher to try and compensate, inter-token latency becomes erratic and head-of-line blocking appears — earlier requests stall while later ones wait. This is not a bug. Ollama was not designed for this workload.

TGI handles moderate concurrency well thanks to its continuous batching implementation. Under heavy load, vLLM pulls ahead significantly, but for workloads with 5-10 concurrent users TGI remains competitive.

Long Context Performance

One area where TGI v3 fights back is long-context workloads. TGI v3 claims 13x faster responses than vLLM on prompts exceeding 200,000 tokens by keeping the initial conversation KV cache around with approximately 5 microsecond lookup overhead. This is a meaningful advantage for applications that maintain long conversation histories. For most other workloads, vLLM's throughput advantage holds.

Multi-GPU and Scaling

This is where Ollama exits the conversation entirely. Ollama runs on a single node. You can put it behind a load balancer with multiple replicas, but each replica is independent — there is no distributed inference.

vLLM supports tensor parallelism and pipeline parallelism across multiple GPUs and multiple nodes using Ray. This means you can shard a 70B model across multiple GPUs on the same machine, or across machines in a cluster. There are production Helm charts, multi-node serving scripts, and Kubernetes guides available. If you are running models that do not fit on a single GPU or need to serve high traffic, this is the path.

TGI supports tensor parallelism via NCCL sharding across GPUs on a single node, but does not support multi-node inference. For single-machine multi-GPU setups it works, but you cannot scale beyond one server.

Model Support and Quantization

All three support the models most developers care about — LLaMA, Mistral, Mixtral, Qwen, Gemma, Phi, and others. The differences are in format support and quantization options.

Ollama uses GGUF exclusively. All models in its registry are GGUF-based, with quantization handled through GGUF quant types like Q4_K_M. You can import Safetensors models via a Modelfile, but it is a manual process.

vLLM has the broadest quantization support: AWQ, GPTQ, GGUF, INT4, INT8, FP8 weight-only quantization, and FP8 KV cache quantization across multiple hardware platforms. It also has first-class LoRA and Multi-LoRA support, which means you can serve multiple fine-tuned adapters from a single base model simultaneously.

TGI supports AWQ, GPTQ, bitsandbytes, EETQ, EXL2, Marlin, and FP8. Its model support is solid, though non-core model architectures fall back to slower transformers code without optimizations.

Observability and Production Readiness

If you are running inference in production, you need to know what is happening inside the engine.

vLLM exposes a rich Prometheus metrics endpoint at /metrics covering request counts, latencies, KV cache usage, queue lengths, and GPU utilization. Grafana dashboard examples and OpenTelemetry integration are documented.

TGI ships with Prometheus metrics and OpenTelemetry tracing built in from day one. This is one of TGI's genuine strengths — Hugging Face built it for their own production infrastructure and the observability reflects that.

Ollama provides basic logs. There is no native Prometheus endpoint, no structured metrics, and no built-in tracing. For local development this is fine. For production monitoring it is a gap you would need to fill yourself.

The Decision Framework

After running these comparisons and deploying all three in different contexts, the decision comes down to two questions: how many concurrent users will hit your model, and how much operational complexity can you handle?

Use Ollama When

Your workload is single-user or low-concurrency. You are prototyping, building a local dev environment, or running a personal assistant. You want to go from zero to a working model in under a minute. You are on macOS, Windows, or Linux and want minimal setup.

Ollama is genuinely excellent at what it does. The developer experience is the best in the category — ollama pull and you are running. The OpenAI-compatible API at localhost:11434/v1/chat/completions integrates cleanly with frameworks like Vercel AI SDK and Autogen. Just do not expect it to scale to production traffic.

Use vLLM When

You are building a production API, an internal platform, or anything that needs to handle concurrent users efficiently. You need multi-GPU or multi-node deployments. You want the best throughput per GPU dollar. You care about long-term ecosystem momentum — vLLM has 70,000+ GitHub stars, extremely active development, and is now the recommended engine from Hugging Face themselves.

The tradeoff is complexity. vLLM's configuration surface is large — batch sizes, cache sizes, parallelism settings — and misconfiguration can underutilize your hardware. CPU-only inference is not a primary target, so if you need to run without a GPU, look elsewhere.

Use TGI When

You already have TGI deployments running in production and they work. The observability and safety features are strong, and if your current workload is stable, there is no urgency to migrate. But for new projects, follow Hugging Face's own guidance and use vLLM or SGLang. TGI is in maintenance mode, which means no new features, limited community investment, and eventual deprecation risk.

Quick Reference

What About llama.cpp and SGLang?

I focused on these three because they represent the most common decision developers face today. But two others deserve mention.

llama.cpp is the C/C++ inference library that Ollama is built on. If you want maximum portability — CPU inference, Apple Silicon, AMD GPUs via Vulkan, edge devices — llama.cpp gives you that directly. It supports 1.5 to 8-bit integer quantization and CPU+GPU hybrid inference for models larger than your VRAM. Many developers use llama.cpp directly when they need fine-grained control that Ollama abstracts away.

SGLang is increasingly mentioned alongside vLLM as a high-performance serving option, particularly for agentic workflows. Hugging Face recommends it as an alternative to vLLM for new deployments. If you are evaluating vLLM, SGLang is worth benchmarking against for your specific workload.

Wrapping Up

The inference engine landscape looks complicated, but the decision is simpler than it appears. Ollama is for local development. vLLM is for production. TGI is for existing deployments that have not migrated yet. The benchmark data consistently supports this division, and Hugging Face's own recommendation to move away from TGI confirms where the ecosystem is heading.

The harder question — whether to self-host at all versus using API providers — depends on your traffic patterns, privacy requirements, and willingness to manage GPU infrastructure. But that is a different article.

Let me know in the comments if you have questions, and subscribe for more practical development guides.

Thanks, Matija