Stay updated on AI infrastructure, inference techniques, and performance optimization.

<p>Subscribe to our <u>newsletter</u></p>

Expert how-tos, deep-dive guides, and real-world stories from the Bento team, to help you build and scale AI at blazing speed.

Sherlock Xu

Explore the best open-source LLMs and find answers to common FAQs about performance, inference optimization, and self-hosted deployment.

best-open-source-llms.png

The Best Open-Source LLMs in 2026

Learn ChatGPT usage limits for Free, Go, Plus, Business, and Pro plans (2026 update). Understand why they exist and how to remove them with self-hosted LLMs.

chatgpt-usage-limits.png

ChatGPT Usage Limits: What They Are and How to Get Rid of Them

Understand the differences among DeepSeek-V3, R1, V3.1, V3.2, V4, and distilled models. Learn how to choose the right model and deploy them securely.

deepseek-models-cover-image-v4.png

The Complete Guide to DeepSeek Models: V3, R1, V4 and Beyond

<p>You load a 70B LLM on 2× NVIDIA A100 80 GB GPUs and think:</p><p><i>“This has more memory than I need. My model should run fine.”</i></p><p>The model loads successfully. But the moment you start generating text, VRAM usage spikes. Suddenly, you hit the Out of Memory (OOM) error.</p><p>Your batch size collapses to 1. Your context length has to be cut in half. Your inference throughput drops to a crawl.</p><p>So you start searching for fixes like KV-aware routing and cache offloading. Every solution sounds promising, but brings new dependencies, complex optimization tricks, and performance trade-offs. Before you know it, you’re buried in GitHub issues and CUDA error logs. This is all because your GPU memory didn’t behave the way the spec sheet said it would.</p><p>It doesn't have to be this hard.</p><p>In this post, we’ll clear up the confusion. You’ll learn:</p><ul><li>What GPU memory is</li><li>How LLMs use GPU memory during inference</li><li>How to calculate memory requirements</li><li>How to optimize memory usage and how our Inference Platform can help you</li></ul><h2>What is GPU memory (VRAM)?</h2><p>When people talk about GPU memory (e.g., A100 80 GB) for inference, they often mean dedicated VRAM (video random access memory). It is high-speed memory physically attached to the GPU chip, such as HBM3 or GDDR6X.</p><p>Key facts:</p><ul><li><strong>Extremely high speed.</strong> Modern <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/nvidia-data-center-gpus-explained-a100-h200-b200-and-beyond">data center GPUs</a> can read and write data at terabytes per second. For example, an NVIDIA H200 reaches up to 4.8 TB/s of memory bandwidth.</li><li><strong>Exclusive to the GPU.</strong> VRAM isn’t shared with the CPU. It’s a dedicated workspace.</li><li><strong>Stores what the model needs at runtime</strong>. That mainly includes model weights, activations, and KV cache.</li><li><strong>Directly impacts LLM inference performance.</strong> VRAM capacity and bandwidth influence throughput, latency, maximum context length, and how many concurrent requests you can serve.</li></ul><p>Note that strictly speaking, GPU memory and VRAM <strong>are not exactly the same thing</strong>.</p><p>GPU memory is a broader term that means whatever memory the GPU is currently using. It usually means VRAM (and people often use the terms interchangeably), but can refer to other sources depending on the system architecture.</p><p>For example, integrated GPUs are the most common case where GPU memory ≠ VRAM. They carve out a portion of system RAM, and there's no dedicated VRAM hardware at all. We’ll break that down in the next section.</p><h2>What is shared GPU memory?</h2><p>Shared GPU memory refers to system RAM that the GPU can use when it runs out of its own dedicated VRAM.</p><ul><li>It’s not physically located on the GPU chip.</li><li>It’s part of the CPU’s memory (DRAM) that’s dynamically allocated to the GPU when needed.</li><li>It’s common in integrated GPUs (like Intel Iris, Apple M-series, or AMD APUs), where GPU and CPU share the same physical memory pool.</li></ul><p>If you’re using data center GPUs like A100 or H200 for LLM inference, <strong>you don’t need to care about shared GPU memory</strong>. They rely entirely on their own dedicated HBM, which is fast and optimized for large-scale AI workloads.</p><p class="p1">Note that the term “shared GPU memory” discussed here is not the same as <strong>shared memory (SMEM)</strong> used in GPU programming, which is a fast, on-chip memory inside each GPU core (SM) for cooperation between threads. If you’re interested in how on-chip memory (registers, shared memory, caches) impacts performance in LLM inference, see the <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/kernel-optimization/gpu-architecture-fundamentals">LLM Inference Handbook</a> for a deeper explanation.</p><h2>How GPU memory is used in LLM inference?</h2><p>GPU memory isn’t just a container for model weights. It’s actively consumed throughout the entire inference process. At a high level, VRAM is used for:</p><ol><li>Loading model weights</li><li>Building and expanding the KV cache during prefill and decode</li><li>CUDA, framework and runtime overhead</li></ol><p>Let’s break down each stage.</p><h3>Model weights</h3><p>Before inference starts, all model weights must be loaded into GPU memory (or sharded across GPUs if using tensor parallelism).</p><p>You can calculate the baseline memory required just to load the model like this:</p><pre><code class="language-bash language-python">Model Memory ≈ num_parameters × bytes_per_parameter</code></pre><p>For example, a 70B model in FP16 precision requires:</p><pre><code class="language-python">70B × 2 = 140 GB</code></pre><blockquote><p>Note: FP32 → 4 bytes, FP16/BF16 → 2 bytes, INT8/FP8 → 1 byte, INT4/FP4 → 0.5 bytes</p></blockquote><p>This is why a single A100 80 GB cannot hold a 70B FP16 model without quantization or multi-GPU sharding.</p><h3>KV cache</h3><p>Once inference begins, memory consumption continues to grow due to the KV cache.</p><p>During prefill, the model processes the full input prompt.</p><ul><li>It builds a KV cache to store the key and value vectors for every token at every layer</li><li>The KV cache grows linearly with the input sequence length.</li><li>Memory usage can increase rapidly for long prompts.</li></ul><p>During decoding, the model generates tokens one by one autoregressively. Each new token appends to the KV cache. Even if the prompt was short, long generations would still expand the KV cache significantly. Learn more about <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/llm-inference-basics/how-does-llm-inference-work">LLM inference in our handbook</a>.</p><p>For chat applications with multi-turn conversations, every new turn must include all previous messages. This means:</p><ul><li>The effective context window grows with each turn</li><li>The KV cache becomes the dominant VRAM consumer</li><li>Eventually, memory fills up even if the model weights fit comfortably</li></ul><figure class="image image_resized" style="width:75%;"><img src="/uploads/llm_context_window_7eab1a7b66.png" alt="llm-context-window.png" srcset="/uploads/xsmall_llm_context_window_7eab1a7b66.png 64w, /uploads/thumbnail_llm_context_window_7eab1a7b66.png 245w, /uploads/small_llm_context_window_7eab1a7b66.png 500w, /uploads/medium_llm_context_window_7eab1a7b66.png 750w, /uploads/large_llm_context_window_7eab1a7b66.png 1000w, /uploads/xlarge_llm_context_window_7eab1a7b66.png 1920w" sizes="100vw" width="1920"></figure><p>This is why models with long context windows require GPUs with large VRAM.</p><h3>Framework + CUDA overhead</h3><p>Even after accounting for weights and KV cache, a few more items consume VRAM:</p><ul><li>CUDA kernels and workspaces</li><li>Framework-specific allocators (e.g., PyTorch fragmentation)</li><li>Runtime overhead from vLLM, TensorRT-LLM, or SGLang</li><li>Temporary activation memory during prefill</li></ul><p>The specific overhead size depends on the framework and GPU.</p><h2>How to calculate the required GPU memory for an LLM</h2><p>A rough estimate for the required GPU memory can be calculated by multiplying the number of parameters by the bytes per parameter and then add extra overhead (%).</p><pre><code class="language-bash language-python">Memory (GB) = num_parameters × bytes_per_parameter × (1 + Overhead)</code></pre><p>For LLM inference, <strong>the real variable you must watch is the KV cache</strong>, not the model weights. This is what grows with context length, batch size, and multi-turn interactions.</p><p><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/getting-started/calculating-gpu-memory-for-llms">Learn more in our handbook</a>.</p><h2>How to calculate the KV cache size</h2><p>The KV cache stores the key/value vectors for every token the model has processed. Its size grows linearly with sequence length, batch size, and number of layers.</p><p>A simplified formula:</p><pre><code class="language-bash language-python">KV Cache Size (GB) =
    2 × batch_size × seq_len × num_layers × hidden_dim × bytes_per_parameter / 1024³</code></pre><p>You can experiment with different batch sizes or context windows using the interactive <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-offloading#how-to-calculate-the-kv-cache-size">KV cache calculator online</a>.</p><h2>Why an LLM “fits on paper” but still runs out of VRAM</h2><p>Before LLMs became common, AI teams were used to serving small NLP or vision models. They were only a few hundred MBs to a few GBs, so a single GPU could easily host multiple models at once with techniques like MIG (Multi-Instance GPU).</p><p>But note that in such cases each model requires very little memory and compute usage is low.</p><p>This led to the expectation that <strong>“If my LLM is 60 GB, an 80 GB GPU should be enough and I might be able to run another small model on the same GPU.”</strong></p><p>However, LLMs behave very differently. They don’t just load weights; they grow at runtime.</p><p>As mentioned in previous sections, even if an LLM’s weights fit comfortably in VRAM, it still needs a large and growing amount of memory for the KV cache. It expands with every token in the prompt and every token generated during decoding.</p><p>This results in the problems people commonly hit:</p><ol><li><strong>You can’t use large context windows.</strong> The KV cache scales linearly with sequence length. A model may load fine, but the moment you process a 4k, 8k, or 32k prompt, VRAM usage explodes.</li><li><strong>Concurrency stays low.</strong> If one request consumes a large amount of KV memory, you simply can’t serve many users at once.</li><li><strong>KV cache hit rate drops</strong>. Without enough room to store cache states, caches are evicted early and the model recomputes previous tokens repeatedly. As a result, your latency and inference cost will increase dramatically.</li></ol><p>Ultimately, if you try inference, you’ll hit the classic OOM issue.</p><p>This is why an “LLM that needs 60 GB for weights” often cannot run reliably on an 80 GB GPU with long prompts, high concurrency, or multi-turn conversations.</p><p>The limiting factor isn’t the weights; it’s the runtime memory growth.</p><h2>How to optimize GPU memory usage</h2><p>Once you understand why LLMs consume so much memory, the next question is: How do you reduce memory pressure and scale?</p><p>Below are the some of the most effective strategies used in production. All of them work, but several require complex engineering to deploy correctly.</p><p>And that’s exactly why we built our inference platform: <strong>to give AI teams all of these optimizations out-of-the-box, without hiring a dedicated infra team.</strong></p><h3>Quantization</h3><p><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/getting-started/llm-quantization">Quantization</a> reduces the precision of model weights (e.g., from FP16 to INT8 or INT4), which lowers memory usage.</p><p>For example, a 70B model in FP16 requires 140 GB for model weights alone, plus additional memory for KV cache and overhead. After INT4 quantization, the weights need 35 GB of VRAM. This is a 4× reduction in memory footprint, enough to run a previously multi-GPU model on a single H200 (141 GB) with room left for KV cache and other overhead.</p><p>Note that quantization does introduce accuracy degradation. However, modern quantization methods like GPTQ and AWQ minimize this loss. For many workloads (chatbots, RAG, internal tools), the gains in memory savings and throughput outweigh the precision trade-off.</p><p><strong>How Bento helps</strong>:</p><p>Bento supports any <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/navigating-the-world-of-open-source-large-language-models">open-source</a>, fine-tuned, or custom model, including quantized formats (INT8, FP8, INT4, GGUF, AWQ, GPTQ, etc.). You can load and serve quantized models directly without extra engineering work, container hacks, or custom inference wrappers.</p><h3>Distributed inference techniques</h3><p>Distributed inference lets you spread the workload across multiple GPUs when a model or context window exceeds the capacity of a single device.</p><h4>Tensor parallelism</h4><p><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism">Tensor parallelism</a> splits the internal computations of each layer across multiple GPUs. Instead of placing entire layers on different devices, it slices the layer's tensors into smaller chunks and distributes them across multiple GPUs.</p><p>This is useful when:</p><ul><li>the model is too large to fit on a single GPU, or</li><li>long-context workloads cause memory usage to exceed a single device’s capacity</li></ul><figure class="image image_resized" style="width:75%;"><img src="/uploads/tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png" alt="tp-inference-6f52474ab788bb99f32150072b34921c.png" srcset="/uploads/xsmall_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 64w, /uploads/thumbnail_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 221w, /uploads/small_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 500w, /uploads/medium_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 750w, /uploads/large_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 1000w, /uploads/xlarge_tp_inference_6f52474ab788bb99f32150072b34921c_e8b519d825.png 1920w" sizes="100vw" width="1920"></figure><p>However, tensor parallelism introduces significant communication overhead, which:</p><ul><li>increases latency</li><li>demands high-bandwidth interconnects</li><li>adds complexity to your deployment setup</li></ul><p>The overhead can become a bottleneck and affect LLM performance if not carefully optimized.</p><h4>KV cache optimizations</h4><p>KV cache is often the real memory bottleneck. You can integrate multiple runtime strategies to control its growth and reduce VRAM usage.</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-caching">Prefix caching</a>: Reuses shared prompts across requests and users (e.g., system prompts). This cuts redundant KV computation and is especially useful for chat applications, RAG systems, and AI agents.</li><li><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-utilization-aware-load-balancing">KV-aware routing</a>: Ensures tokens for the same request are executed on the same GPU or worker, preventing fragmented KV allocations and improving the cache hit rate. In our internal tests with long context data (&gt;20K), KV-aware routing delivered:<ul><li>12× higher input throughput</li><li>3.5× higher output throughput</li><li>4× lower TTFT</li><li>90%+ prefix cache hit rates</li></ul></li><li><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-offloading">KV cache offloading</a>: Moves less frequently used KV blocks to lower-cost storage like CPU memory or disk, freeing up VRAM without blocking token generation.</li></ul><p>&nbsp;</p><hr><p>&nbsp;</p><p><strong>These strategies work, but they are hard to implement yourself.</strong> To configure all of them manually, you’d need to handle:</p><ul><li>multi-GPU communication and synchronization</li><li>Custom CUDA kernels or inference backends</li><li>routing logic across workers to maximize cache reuse</li><li>KV cache eviction and offloading logic</li></ul><p>Most AI teams don’t have the engineering bandwidth to build and maintain this infrastructure.</p><p><strong>How Bento helps</strong>:</p><p>Bento packages these optimizations into a production-ready inference platform so you don’t have to reinvent the entire inference stack:</p><ul><li>Automatic distributed inference with frameworks like MAX, vLLM, and SGLang</li><li>Built-in KV-aware routing, offloading, and prefix caching</li><li>Running models with these optimizations on any cloud (including BYOC) or on-prem cluster</li></ul><p>Instead of spending time building your own inference system, you get an optimized one that handles GPU memory intelligently. With it, your models run faster, cheaper, and with higher throughput, and your engineering team can focus on AI development.</p><h2>Conclusion</h2><p>VRAM is far more than a static storage specification. It is a dynamic resource that fluctuates with every token generated, every new user connected, and every millisecond of your inference runtime. Understanding the math behind model weights and KV cache growth is the first step to avoiding those deployment-day crashes.</p><p>The next step, namely solving it with quantization and different optimization techniques, is an engineering marathon.</p><p>That is why we built Bento. We believe your team should focus on building better AI products, not fighting with infrastructure issues.</p><p>If you want to run LLMs reliably, efficiently, and at scale:</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://forum.modular.com/c/bento/31?utm_source=bentoml_blog">Join our community forum</a> to learn from thousands of AI engineers</li><li><a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/request-demo?utm_source=bentoml_blog">Schedule a call with our team</a> to explore your deployment needs</li><li><a target="_blank" rel="noopener noreferrer" href="https://console.modular.com/signup?utm_source=bentoml_blog">Sign up for our Inference Platform</a> and start serving LLMs today</li></ul>

A complete guide to GPU memory for LLMs: VRAM, KV cache, context windows, quantization, parallelism, and inference optimizations for faster, more efficient inference.

gpu-memory-vram.png

What is GPU Memory and Why it Matters for LLM Inference

Explore the top open-source TTS models and find answers to some FAQs about them.

tts-models.png

The Best Open-Source Text-to-Speech Models in 2026

Explore top open-source image generation models and find answers to FAQs about them.

image-generation-models.png

The Best Open-Source Image Generation Models in 2026

Small language models (SLMs) are compact LLMs designed to run efficiently in resource-constrained environments. They are now good enough for many production workloads.

slms-blog-cover-image.png

The Best Open-Source Small Language Models (SLMs) in 2026

Chaoyu Yang

This guide helps you match specific LLM inference bottlenecks to the highest-impact optimization strategies, and understand when to implement each one as your workloads evolve.

inference-optimizations-cover-image-blog.png

6 Production-Tested Optimization Strategies for High-Performance LLM Inference

This guide shows enterprise teams how to identify hidden trade-offs in LLM deployment and evaluate performance through the lens of your actual workloads, not simplified metrics.

balance-speed-cost-and-quality-in-llm-inference.png

Beyond Tokens-per-Second: How to Balance Speed, Cost, and Quality in LLM Inference

This guide gives enterprise leaders a clear view of the infrastructure trends and architecture patterns shaping modern AI systems.

ai-infra-trends.png

Emerging Trends in AI Infrastructure and How Enterprise Teams Can Stay Ahead

<p>Running large reasoning models no longer means relying on third-party APIs. With <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/navigating-the-world-of-open-source-large-language-models">open-source models</a> like DeepSeek-R1 and gpt-oss, you can now self-host powerful reasoning models and build your private inference API.</p><p>Unlike closed-source APIs that operate as black boxes, open-source models let you customize inference logic, serve them efficiently with frameworks like vLLM, and apply advanced techniques such as prefill–decode disaggregation for optimized performance. The result: more control, flexibility, and lower cost.</p><p>In this post, you’ll learn how to self-host gpt-oss using vLLM and BentoML. We will deploy it to BentoCloud, our fully managed inference platform with fast autoscaling, LLM-specific observability, and production-grade security built in.</p><h2>What is vLLM?</h2><p>vLLM is a fast and efficient open-source library designed for LLM inference and serving. Developed by researchers at UC Berkeley, vLLM stands out for its high-performance capabilities in handling LLMs. It features advanced techniques such as continuous batching, speculative decoding, disaggregated prefilling, and automatic prefix caching.</p><h2>What is gpt-oss?</h2><p>gpt-oss is an open-source reasoning model developed by OpenAI. It comes in two main variants:&nbsp;<code>gpt-oss-120B</code>&nbsp;and&nbsp;<code>gpt-oss-20B</code>. Trained using reinforcement learning and insights from OpenAI’s frontier models like o3, gpt-oss is good at complex reasoning tasks. It’s also practical for general use cases.</p><p>gpt-oss is used as an example in this guide. You can choose any open-source model based on your needs. Learn more about the <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/navigating-the-world-of-open-source-large-language-models">best open-source LLMs in 2026</a>.</p><h2>Set up the environment</h2><p>I suggest you create a virtual environment to keep your dependencies organized:</p><pre><code class="language-bash">python -m venv vllm-gpt-oss
source vllm-gpt-oss/bin/activate</code></pre><p>Next, clone the <a target="_blank" rel="noopener noreferrer" href="https://github.com/bentoml/BentoVLLM">BentoVLLM repo</a> and install the dependencies:</p><pre><code class="language-bash">git clone https://github.com/bentoml/BentoVLLM.git
cd BentoVLLM/gpt-oss-20b
pip install -r requirements.txt</code></pre><h2>Build a gpt-oss inference API with vLLM and BentoML</h2><p>Everything you need is in the repo cloned. Before deploying to the cloud, let’s go through the key code implementations.</p><h3>Define model and GPU configurations</h3><p>First, specify the model and GPU. In this example, we’re using gpt-oss-20b with a single NVIDIA H100 GPU, but you can switch to any LLM supported by vLLM.</p><pre><code class="language-python">import pydantic
import bentoml

# Use Pydantic to validate data
class BentoArgs(pydantic.BaseModel):
  name: str = 'gpt-oss-20b'
  gpu_type: str = 'nvidia-h100-80gb' # GPU type on BentoCloud
  tp: int = 1 # One GPU here for tensor parallelism
  model_id: str = 'openai/gpt-oss-20b'
  port: int = 8000
  # Other optional fields omitted for brevity

# Make the args injectable
bento_args = bentoml.use_arguments(BentoArgs)</code></pre><p>Here we use <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/build-with-bentoml/template-arguments.html">template arguments</a> from BentoML to inject dynamic and validated parameters at serve, build, and deploy time. These arguments can be referenced just like normal Python variables.</p><h3>Configure the runtime environment</h3><p>BentoML allows you to package your code, dependencies, and model references into a unified Bento artifact, which simplifies deployment across different environments.</p><p>Here’s how you define the <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/build-with-bentoml/runtime-environment.html">runtime environment</a> for your model using a Bento image:</p><pre><code class="language-python">image = bentoml.images.Image(python_version="3.12") \
    .system_packages("curl", "git") \
    .requirements_file("requirements.txt")</code></pre><p>For AMD GPU support, you can customize the image setup as follows:</p><pre><code class="language-python">image.base_image = 'rocm/vllm:rocm6.4.1_vllm_0.10.1_20250909'
# Disable locking of Python packages for AMD GPUs to exclude nvidia-* dependencies
image.lock_python_packages = False
# The GPU device is accessible by group 992
image.run('groupadd -g 992 -o rocm &amp;&amp; usermod -aG rocm bentoml &amp;&amp; usermod -aG render bentoml')
# Remove the vllm and torch deps to reuse the pre-installed ones in the base image
image.run('uv pip uninstall vllm torch torchvision torchaudio triton')</code></pre><h3>Launch a vLLM server within a BentoML Service</h3><p>Set up a vLLM server that listens on port 8000 and serves requests to our model.</p><pre><code class="language-python">class LLM:
  # Download the model weights from HF, skipping large checkpoints
  hf_model = bentoml.models.HuggingFaceModel(bento_args.model_id.lower(), exclude=[".pth", ".pt", "original/**/*"])

  def __command__(self) -&gt; list[str]:
    return [
      'vllm',
      'serve',
      self.hf_model,
      '--port',
      str(bento_args.port),
			# ...extra CLI args
      '--served-model-name',
      bento_args.model_id,
    ]</code></pre><p>Inside the class, we use the <code>HuggingFaceModel</code> API to load model weights from Hugging Face. Traditionally, model loading can be time-consuming, especially for large models.</p><p>BentoML optimizes this process by preloading models during image build instead of at service startup. The downloaded weights are cached and mounted directly into the container at runtime, which reduces cold start latency and accelerates scaling on BentoCloud. This mechanism ensures faster deployments and smoother autoscaling for large models like gpt-oss. Learn more about <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/25x-faster-cold-starts-for-llms-on-kubernetes">LLM cold starts</a>.</p><p>Next, use the&nbsp;<code>@bentoml.service</code>&nbsp;decorator to wrap the class, marking it as a <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/build-with-bentoml/services.html">BentoML Service</a>. This starts vLLM as the serving backend within the Bento.</p><pre><code class="language-python">@bentoml.service(
  name=bento_args.name,
  image=bento_args.image,
  traffic={'timeout': 300},
  resources={'gpu': bento_args.tp, 'gpu_type': bento_args.gpu_type},
)
class LLM:
 ...</code></pre><p>This basic setup is all you need to get your gpt-oss model running with vLLM and BentoML. For more advanced configurations like KV cache tuning, refer to the <a target="_blank" rel="noopener noreferrer" href="https://github.com/bentoml/BentoVLLM/blob/main/gpt-oss-20b/service.py">complete source code</a>. BentoML supports full customization to tailor inference for your use case.</p><h2>Deploy gpt-oss to BentoCloud</h2><p>BentoCloud provides fast and scalable infrastructure for building and scaling AI applications.</p><p>Install BentoML and log in to BentoCloud from the CLI. If you don’t have an account yet, <a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/signup-from-blog-vllm-llm">sign up here for free</a>.</p><pre><code class="language-bash language-python">pip install bentoml
bentoml cloud login</code></pre><p>Create a secret to store your HF token and reference it when running the deployment command:</p><pre><code class="language-bash language-python">bentoml secret create huggingface HF_TOKEN=$HF_TOKEN

bentoml deploy --secret huggingface</code></pre><p>Once it is up and running, go to the Deployment details page, where it displays the exposed <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/llm-inference-basics/openai-compatible-api">OpenAI-compatible API</a>. As you interact with it, you’ll see real-time <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/llm-inference-metrics">inference metrics</a> such as tokens per second and Time to First Token (TTFT).</p><h2>Test the OpenAI-compatible API</h2><p>You can call the endpoint directly with any OpenAI client. Just set the <code>base_url</code> to your BentoCloud Deployment URL.</p><pre><code class="language-python">from openai import OpenAI

client = OpenAI(base_url='your_deployment_endpoint', api_key='na')

# Use the following func to get the available models
# client.models.list()

chat_completion = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {
            "role": "user",
            "content": "Who are you? Please respond in pirate speak!"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    # Extract and print the content of the model's reply
    print(chunk.choices[0].delta.content or "", end="")</code></pre><h2>Enable scale-to-zero</h2><p>By default, each Deployment runs a single replica. To optimize cost, set minimum replicas to 0 so it scales down during idle time and scales up automatically when new requests arrive.</p><pre><code class="language-bash language-python">bentoml deployment update deployment_name --scaling-min 0 --scaling-max 3</code></pre><h2>Run gpt-oss locally</h2><p>If you have a powerful GPU available, you can serve gpt-oss locally using BentoML.</p><p>For better performance on NVIDIA GPUs, we recommend installing the FlashInfer library before serving:</p><pre><code class="language-bash language-python">pip install flashinfer-python --extra-index-url https://flashinfer.ai/whl/cu124/torch2.6
export HF_TOKEN=&lt;your-api-key&gt;
bentoml serve</code></pre><p>The server will start at <a target="_blank" rel="noopener noreferrer" href="http://localhost:3000">http://localhost:3000</a>. You can send requests directly to your local vLLM + BentoML service through any OpenAI client.</p><p>To package your project for deployment or portability, build a Bento and containerize it into a Docker image:</p><pre><code class="language-bash language-python">bentoml build
bentoml containerize</code></pre><p>This lets you run the same gpt-oss setup on any infrastructure.</p><h2>More on BentoML and vLLM</h2><p>To go deeper into LLM inference, model serving, and deployment best practices, check out the following resources:</p><ul><li>[Doc] <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/examples/vllm.html">vLLM inference</a></li><li>Read our <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/">LLM Inference Handbook</a> to learn more about inference optimization and LLM deployment techniques.</li><li><a target="_blank" rel="noopener noreferrer" href="https://forum.modular.com/c/bento/31?utm_source=bentoml_blog">Join our community forum</a> to connect with other AI engineers and get help.</li><li><a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/request-demo?utm_source=bentoml_blog">Talk to us</a> about enterprise LLM deployment, BYOC, or on-prem solutions.</li><li><a target="_blank" rel="noopener noreferrer" href="https://console.modular.com/signup?utm_source=bentoml_blog">Sign up for our inference platform</a> to deploy your private gpt-oss model today with one command.</li></ul><p>&nbsp;</p><hr><h2>FAQs</h2><h3>What is BentoCloud?</h3><p><a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/introducing-bentocloud">BentoCloud</a> is a fully managed inference platform for deploying and scaling AI models in production. It provides features like GPU autoscaling, built-in observability, <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/scale-with-bentocloud/deployment/sandboxes.html">Sandboxes</a>, and <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/accelerate-ai-application-development-with-bentoml-codespaces">Codespaces</a> for fast and cost-efficient inference.</p><p>You can deploy gpt-oss or any other model with one command, without any Kubernetes or infrastructure setup.</p><p>BentoCloud is part of the Bento Inference Platform, which also provides <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/infrastructure-and-operations/on-prem-llms">on-prem</a> and <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/infrastructure-and-operations/bring-your-own-cloud">BYOC (Bring Your Own Cloud)</a> solutions.</p><p>BentoCloud is ideal for AI teams that want to prototype quickly without infrastructure overhead. For teams needing more control or strict data governance, <a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-vllm-llm">contact us</a> to explore Bento On-Prem or BYOC.</p><h3>Why use vLLM for gpt-oss?</h3><p>vLLM is one of the fastest open-source inference engines for LLMs. It supports <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/static-dynamic-continuous-batching">continuous batching</a>, <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-caching">prefix caching</a>, and <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation">prefill–decode disaggregation</a>, all of which improve throughput and reduce latency when serving gpt-oss at scale.</p><h3>What is the difference between running gpt-oss with vLLM and Ollama?</h3><p>vLLM is a high-performance inference engine optimized for server-side deployment and large-scale workloads. It supports distributed GPU setups, OpenAI-compatible APIs, and advanced batching techniques.</p><p>Ollama, on the other hand, is a local model runner designed for desktops and small servers. It’s great for lightweight experimentation but not ideal for production-scale serving or multi-GPU deployments.</p><p>If you need speed, scalability, and control in production environments, vLLM + BentoML is the better choice.</p><h3>Can gpt-oss generate images?</h3><p>No. gpt-oss is a language model focused on reasoning and text generation, not image synthesis.</p><p>If you want to generate images, you can integrate gpt-oss with a <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/a-guide-to-open-source-image-generation-models">text-to-image model</a> such as Stable Diffusion and FLUX. BentoML supports deploying these multimodal pipelines together, so you can combine reasoning and generation in one API.</p>

Run OpenAI's gpt-oss model with vLLM and BentoML. Learn to build a fast, private reasoning API and deploy it on BentoCloud with autoscaling.

run-gpt-oss-vllm.png

Deploying OpenAI's gpt-oss model with vLLM and BentoML

Explore the top open-source VLMs and find answers to some FAQs about them.

vision-language-models.png

Multimodal AI: The Best Open-Source Vision Language Models in 2026

BentoML Team

<p>We’re excited to share that BentoML has joined <a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/">Modular</a>! After months of collaborating on customer deployments together, we’re thrilled to officially become one team, as part of a strategic product acquisition.</p><p>This partnership strengthens our ability to support the BentoML project and community. Modular shares our commitment to open source and making AI infrastructure accessible. Together, we’ll be able to push the platform further with capabilities that weren’t possible before.</p><h2>Why this partnership makes sense</h2><p>BentoML’s mission has always been to help teams run AI inference in production without compromising on speed, cost, or control. We’ve built a multi-cloud inference platform that gives teams flexibility over where and how they deploy AI.</p><p>Modular is building a unified AI compute stack, from the <a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/mojo">Mojo programming language</a> to the <a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/max">Max inference engine</a>, that enables high-performance inference across diverse hardware. BentoML brings the cloud layer: the tooling teams rely on to deploy, operate, and scale inference across real-world environments, including multi-cloud and BYOC setups.</p><p>Inference systems are complicated. They have many layers, and getting everything right across the stack is hard. With vertically integrated infrastructure, teams can optimize end-to-end and achieve better performance faster.</p><h2>What this means for you</h2><h3>For BentoML customers</h3><p>Our focus remains the same. We will continue to support production inference across multi-cloud, BYOC, and security-sensitive environments. Your systems will keep running without disruption, with new capabilities coming as we integrate deeper with Modular’s compute stack.</p><h3>For our open-source community</h3><p>BentoML remains Apache 2.0. Our commitment to open source does not change. We’ll continue supporting the workflows and community that have grown around BentoML.</p><p>We’re also hosting an <strong>AMA with Chris Lattner (Modular CEO) and Chaoyu Yang (BentoML founder) on Tuesday, February 17th from 9:30am-11:30am PT</strong> in the <a target="_blank" rel="noopener noreferrer" href="https://forum.modular.com/t/ask-us-anything/2706/1">Modular Forum</a>, where we'll share more details and answer your questions.</p><figure class="image"><img alt="ama-bento-modular.png" src="/uploads/ama_bento_modular_0419a45f47.png" srcset="/uploads/xsmall_ama_bento_modular_0419a45f47.png 64w, /uploads/thumbnail_ama_bento_modular_0419a45f47.png 245w, /uploads/small_ama_bento_modular_0419a45f47.png 500w, /uploads/medium_ama_bento_modular_0419a45f47.png 750w, /uploads/large_ama_bento_modular_0419a45f47.png 1000w, /uploads/xlarge_ama_bento_modular_0419a45f47.png 1920w" sizes="100vw" width="1920"></figure><h2>Looking ahead</h2><p>Inference is shifting from an operational detail to a strategic capability. Teams that can optimize across the full stack and deploy across hardware and clouds can push performance limits for their workloads, deploy where it makes business sense, and adapt their infrastructure as needs evolve.</p><p>Joining Modular allows us to push further in that direction, alongside the customers and community building with us.</p><p>If you want to learn more:</p><ul><li>Check out what Modular is building at <a target="_blank" rel="noopener noreferrer" href="http://modular.com/">modular.com</a>.</li><li>We are <a target="_blank" rel="noopener noreferrer" href="https://www.modular.com/company/careers">actively hiring</a> for roles building the <a target="_blank" rel="noopener noreferrer" href="http://modular.com/deployment">Modular Cloud</a> and more.</li><li>Interested in seeing the Modular Stack in action for your workload? <a target="_blank" rel="noopener noreferrer" href="http://modular.com/deployment">Let’s chat!</a></li></ul><p>Thank you for trusting BentoML and helping shape what we build. We’re grateful to our customers, open-source community, investors, and team for making this next chapter possible.</p>

BentoML is joining Modular to build the next generation of AI inference infrastructure.

large_Modular-Bento.png

medium_Modular-Bento.png

small_Modular-Bento.png

thumbnail_Modular-Bento.png

xlarge_Modular-Bento.png

xsmall_Modular-Bento.png

Modular-Bento.png

BentoML Is Joining Modular

🎉 BentoML is now part of Modular. [Read the announcement.](https://www.modular.com/blog/bentoml-joins-modular)

Bento Blog