Serve an Open-Source LLM at Scale with vLLM on a Rented GPU Instance

What You'll Build

You'll deploy Llama 3.1 8B behind vLLM's OpenAI-compatible API on a rented GPU instance, then verify that continuous batching actually delivers an order-of-magnitude throughput advantage over sequential serving.

Prerequisites

• A cloud GPU instance with at least one NVIDIA A10G (24 GB VRAM). Lambda Labs (~$0.80/hr for A10), RunPod, and Vast.ai all work. An A100 40/80 GB handles larger models or higher concurrency.
• Ubuntu 22.04, CUDA 12.1+ (standard on ML-optimized images). Confirm with nvidia-smi.
• Python 3.10 or 3.11. vLLM 0.6.x is not fully validated on 3.12 yet.
• A Hugging Face account with the Meta Llama 3.1 license accepted at hf.co/meta-llama/Meta-Llama-3.1-8B-Instruct, plus a read-scoped access token.
• SSH access to the instance.

GPU driver installation and VPC networking are out of scope here.

1. Prepare the VM

Verify the GPU and CUDA runtime are visible before touching Python:

nvidia-smi
python3 --version

Create an isolated environment:

python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install --upgrade pip

2. Install vLLM

pip install vllm

This pulls PyTorch 2.4+ compiled for CUDA 12.1 along with vLLM's paged-attention kernels. Expect 3-5 minutes and roughly 4 GB of downloads. Confirm the install:

python -c "import vllm; print(vllm.__version__)"

If your image has CUDA 11.8 (uncommon on current offerings), vLLM 0.6.x no longer supports it. Use the official vLLM Docker image instead: docker pull vllm/vllm-openai:latest.

3. Authenticate with Hugging Face

export HF_TOKEN="hf_YOUR_TOKEN_HERE"

Add this to ~/.profile for persistence, or store it in your cloud provider's secrets manager. Never commit tokens to source control or bake them into Docker layers.

4. Start the Inference Server

vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --served-model-name llama3

Key flags at a glance:

Weights download on first run (~16 GB for BF16). Subsequent starts read from ~/.cache/huggingface. The server is ready when you see:

INFO:     Application startup complete.

5. Send Your First Request

From a second terminal (no venv needed for curl):

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "What is PagedAttention?"}],
    "max_tokens": 150,
    "temperature": 0.7
  }'

The response schema is identical to OpenAI's. Point any OpenAI SDK at the server by changing base_url:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="ignored")
response = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Write a haiku about GPU memory."}],
    max_tokens=60,
)
print(response.choices[0].message.content)

api_key can be any non-empty string by default. See step 7 for enforcing real authentication.

Advertisement

Shadow GPS — know where it is, always Real-time GPS tracking for vehicles, gear and loved ones. No monthly contracts.

6. Load Test: Concurrent Requests

vLLM's continuous batching combines in-flight requests into a single forward pass on every scheduling step, rather than waiting to fill a static batch. The throughput difference is dramatic. Run this to see it:

# bench.py
import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="ignored")
PROMPT = "Summarize the history of neural networks in three sentences."
CONCURRENCY, TOTAL = 50, 200

async def one_request(sem):
    async with sem:
        t0 = time.monotonic()
        r = await client.chat.completions.create(
            model="llama3",
            messages=[{"role": "user", "content": PROMPT}],
            max_tokens=100,
        )
        return time.monotonic() - t0, r.usage.completion_tokens

async def main():
    sem = asyncio.Semaphore(CONCURRENCY)
    t_start = time.monotonic()
    results = await asyncio.gather(*[one_request(sem) for _ in range(TOTAL)])
    elapsed = time.monotonic() - t_start
    total_tok = sum(t for _, t in results)
    print(f"Requests: {TOTAL}  |  Wall time: {elapsed:.1f}s")
    print(f"Aggregate throughput: {total_tok / elapsed:.0f} tokens/sec")
    print(f"Avg latency: {sum(l for l, _ in results) / TOTAL:.2f}s")

asyncio.run(main())

pip install openai
python bench.py

A single A10G running Llama 3.1 8B in BF16 typically reaches 1,200-2,000 aggregate output tokens/sec under this load. Sequential, one-at-a-time serving on the same hardware delivers roughly 35-45 tokens/sec, because the GPU sits idle between requests while waiting for the next one to arrive.

7. Production Hardening

Run vLLM as a systemd service so it survives SSH disconnects:

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM inference server
After=network.target

[Service]
Type=simple
User=ubuntu
Environment="HF_TOKEN=hf_YOUR_TOKEN_HERE"
Environment="PATH=/home/ubuntu/vllm-env/bin:/usr/bin:/bin"
ExecStart=/home/ubuntu/vllm-env/bin/vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --served-model-name llama3
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

The server binds to 127.0.0.1 here, not 0.0.0.0. Never expose port 8000 directly to the internet without authentication. The simplest secure access pattern from a dev machine:

ssh -L 8000:localhost:8000 ubuntu@your-gpu-host

For production, put nginx in front with TLS termination and auth_basic or a JWT validation block, or use Caddy with an API key middleware.

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
sudo journalctl -fu vllm

Verify It Works

Query the models endpoint to confirm the server registered correctly:

curl http://localhost:8000/v1/models | python3 -m json.tool

Expected output contains "id": "llama3" and "object": "model". The scheduler logs in journalctl show live KV cache utilization and per-step batch sizes. Watch nvidia-smi dmon in a separate pane to confirm the GPU is saturated during the load test.

Troubleshooting

torch.cuda.OutOfMemoryError or CUDA out of memory on startup. The KV cache allocation exceeded available VRAM. Lower --gpu-memory-utilization to 0.80, or reduce --max-model-len (halving it roughly halves KV cache size). A 128k context at BF16 needs more than 24 GB on its own.

High latency at low concurrency. Continuous batching optimizes for aggregate throughput, not time-to-first-token on isolated requests. For latency-sensitive single-request workloads, set --max-num-seqs 1 to disable multi-request batching.

OSError: You are trying to access a gated repo. Either HF_TOKEN is not set in the current shell (echo $HF_TOKEN to verify), or your account has not accepted the Llama 3.1 license on Hugging Face. The acceptance must be done on the model page, not just in account settings.

Garbled or truncated outputs. Check that --max-model-len is larger than your prompt token count plus max_tokens. vLLM will silently truncate the prompt from the left when the combined length exceeds the configured limit.

Next Steps

• Quantization: --quantization fp8 runs on-the-fly FP8 quantization (vLLM 0.5+) and fits larger models into the same VRAM. For AWQ, you need a pre-quantized checkpoint from a community hub like Hugging Face; then pass --quantization awq pointing at that repo.
• Multi-GPU tensor parallelism: --tensor-parallel-size 2 on a dual-A100 node gives near-linear throughput scaling with no code changes.
• Structured outputs: --guided-decoding-backend outlines enforces JSON Schema constraints on generation, useful for tool-calling pipelines.
• Metrics: vLLM exposes a Prometheus-compatible endpoint at GET /metrics. Scrape it with a Grafana agent to track KV cache hit rates, queue depth, and inter-token latency percentiles in real time.