Your own inference with vLLM

Run your own OpenAI- and Claude-compatible inference endpoint with vLLM - on your own GPU server, with no per-token cost. DollarDeploy now ships a native vLLM template: it runs under systemd, straight on the host, with no Docker layer in the way. Connect a Verda GPU server, pick a model, and you get a /v1/chat/completions endpoint that any OpenAI or Anthropic SDK can talk to.

We built this template, deployed it on a live Verda H200, and verified it end to end before writing this post. Below is exactly what runs, what hardware you need for which model, and how to ship it yourself.

Why native vLLM instead of Docker

Most "run an LLM" guides hand you a Docker Compose file. That works, but on a GPU box it adds a container runtime, an image layer, and a second copy of the CUDA userspace between your model and the metal. Our vLLM template skips all of it:

• Runs as a systemd service, directly on the host. Faster to start, easier to inspect with journalctl, one less moving part.
• Dependencies via uv from a plain pyproject.toml. uv auto-selects the PyTorch CUDA wheels that match the host's driver, so you don't install a CUDA toolkit.
• OpenAI and Claude compatible. The endpoint speaks both /v1/chat/completions (OpenAI) and /v1/messages (Anthropic), so it's a drop-in base URL for the OpenAI SDK, the Anthropic SDK, Cursor, opencode, or Claude Code.

DollarDeploy handles the parts you don't want to: build, deploy, HTTPS via Let's Encrypt, logs, monitoring, and instant rollback - all on by default.

Why self-host inference at all

Three reasons, same as they've always been:

• Fixed cost instead of a metered bill. A rented GPU costs the same whether it serves 100 tokens or 100 million tokens. There's no per-token line item that scales with your demand. You compare a known hourly rate against a usage-based API invoice - and at real volume, the GPU wins.
• Your data stays on your server. Prompts and completions never leave a box you control - and you pick the jurisdiction it runs in, EU included, so residency is your call, not a vendor's. No third-party retention policy to read, no data-processing addendum to negotiate.
• No shared-tenant surprises. A dedicated GPU gives you consistent latency and no rate limits but the ones you set.

This is the same self-hosted, own-your-infrastructure story DollarDeploy tells for web apps - total cost typically 10-30% of a hyperscaler once you're past a trial - applied to inference.

What the template runs

Out of the box the template serves Qwen/Qwen3.8-27B-FP8 - a 27B model with a 256K context window, native tool calling, and hybrid linear/full attention. We run it in FP8: we don't quantize models ourselves, we deploy the FP8 checkpoints the model authors publish, so the weights you serve are the weights that were tested.

On our verification run - a single Verda H200 141GB, one GPU, TENSOR_PARALLEL_SIZE=1 - Qwen3.8-27B-FP8 loaded in about 30 GB of VRAM and served at ~90 tokens/sec on a single stream (batched throughput is far higher). Swapping models is two environment variables: MODEL_NAME and its matching MODEL_CALL_PARSER.

One endpoint, many callers

Concurrency is the reason a single GPU is worth renting. vLLM uses continuous batching and PagedAttention to pack many in-flight requests onto the same card, instead of running them one after another. On the H200 the model's ~30 GB footprint leaves over 100 GB of VRAM for the KV cache - and the KV cache, not the weights, is what serves parallel requests. Live sequences times context length is what fills it, so that headroom is what you are actually buying.

In practice one shared endpoint on a single H200 serves several customers or apps at once with room to spare. You only reach for a second GPU when the model gets bigger or you are into hundreds of concurrent users, not a handful. Hand every customer the same base URL and give each one its own API key; put a proxy like LiteLLM in front if you want per-key metering.

Coming from Ollama or a GGUF file on llama.cpp? Those are tuned for one request at a time and for squeezing a model onto a small card - great on a laptop, wrong for serving. When a real datacenter GPU does the work and several callers share it, an FP8 checkpoint on vLLM serves far more throughput than the same model as a GGUF.

Sizing: which Verda instance for which model

FP8 weights are roughly one byte per parameter, so the checkpoint size tells you the VRAM floor. Add headroom for the KV cache and activations. Because we don't re-quantize, these are the real published FP8 checkpoint sizes:

Prices are Verda on-demand, USD, per instance (live pricing); spot is roughly half. You pay only for the hours the server runs.

Notes:

• 1x H200 is the accessible entry point. A single Hopper-class card comfortably runs a 27B FP8 model with room for a large context - the setup we actually tested.
• The frontier MoE models are large. GLM-5.x FP8 checkpoints are ~750 GB, so they need a full 8-GPU node (8x H200 = 1,128 GB, or 8x B200 = 1,440 GB). Kimi K3 at ~1.5 TB spans two nodes. Set TENSOR_PARALLEL_SIZE to the GPU count and the template does the rest.
• Use an FP8-native GPU. FP8 wants Hopper (H200) or Blackwell (B200/B300). Ampere (A100) can load FP8 via fallback kernels but isn't native - pick H200 or newer.

Deploy it

You need a Verda account connected to DollarDeploy, and a HuggingFace token (HF_TOKEN) for model downloads.

From the dashboard: open the template library, pick vLLM (native), choose a GPU host to provision on Verda (start with 1x H200), set HF_TOKEN, and deploy. First boot takes ~15 minutes while it installs dependencies and downloads the model (~27 GB). In the last few minutes vLLM compiles and warms up before it binds the port, so the health check reports not-ready (a 000 in the logs) until it comes up - that is the normal cold start, not a failure.

From the CLI (ddc):

# provision a single-H200 host on Verda and deploy the template to it
ddc app deploy --template vllm \
  --create-host --provider verda \
  --type 1H200.141S.44V --region FIN-03 \
  --env HF_TOKEN=hf_your_token_here

The app comes up with HTTPS, a generated API key, and a small landing page showing your base URL and copy-paste snippets.

Use it

The endpoint is OpenAI-compatible, so point any client at https://<your-app>/v1 with your API key:

curl https://<your-app>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3.8-27B-FP8",
    "messages": [{"role": "user", "content": "Write a haiku about GPUs."}]
  }'

from openai import OpenAI

client = OpenAI(base_url="https://<your-app>/v1", api_key="YOUR_VLLM_API_KEY")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B-FP8",
    messages=[{"role": "user", "content": "Explain tensor parallelism in one sentence."}],
)
print(resp.choices[0].message.content)

Because it also speaks the Claude Messages API, you can drive it from Claude Code directly:

export ANTHROPIC_BASE_URL="https://<your-app>"
export ANTHROPIC_AUTH_TOKEN="YOUR_VLLM_API_KEY"
export ANTHROPIC_MODEL="Qwen/Qwen3.8-27B-FP8"
claude

The cost trade

A single Verda H200 is $4.33/hour on-demand - about $3,118/month if you leave it running 24/7, or roughly $1,560 on spot. But you only pay for the hours the server is up: a job that runs a few hours a day costs a fraction of that, and you can shut it down between runs. Whatever the shape, it's a fixed number you can plan around - the GPU costs the same whether it serves ten requests or ten million.

A metered API bills the opposite way: the invoice climbs with every token, and a viral moment turns into a four-figure surprise. Self-hosting a 27B model pays off once you're sustaining real throughput (an H200 batches well past a thousand tokens/second) - or the moment your prompts simply can't leave your own servers. Check Verda's live pricing for current rates and size to the table above.

That's the whole pitch: your model, your GPU, your endpoint - with the build, HTTPS, and monitoring handled, and no invoice surprise at the end of the month.

Deploy the vLLM template

Spin up your own Verda inference endpoint in minutes: Deploy vLLM ->

Questions, issues, or feedback? Email hello@dollardeploy.com.