`vllm://`¶

vLLM — self-hosted, high-throughput inference server with an OpenAI-compatible REST API. The standard way to serve open-weights models in production with paged attention, continuous batching, and speculative decoding.

Built on pkg/component/llmutil/ — same chat/embeddings shape as openai://. This page covers only the vLLM-specific defaults.

URI shape¶

vllm://chat?model=<served-model-name>&baseURL=http://<host>:<port>/v1&temperature=<float>
vllm://embeddings?model=<served-model-name>&baseURL=<url>

Parameters¶

Param	Type	Default	Producer/Consumer/Both	Description
path	enum	required	Producer	`chat` or `embeddings` (depending on the model loaded by vLLM).
`model`	string	`default`	Producer	The model name the vLLM server advertises (set via the `--served-model-name` flag at server startup). Use `default` if you didn't override it.
`apiKey`	string	(empty)	Producer	Optional bearer token if the vLLM server runs with `--api-key`.
`baseURL`	string	`http://localhost:8000/v1`	Producer	URL of the vLLM server's OpenAI-compatible endpoint.
`temperature`	float	`0.7`	Producer	Sampling temperature.
`maxTokens`	int	`4096`	Producer	Output cap.

Producer-only. Default timeout 300 s (longer than other providers because cold-loading a multi-billion-parameter model is slow; once warm, vLLM is fast).

Headers¶

Namespaced under vllm.* (vllm.model, vllm.id, vllm.finishReason, vllm.usage.*). Input header systemPrompt works the same way.

Examples¶

Talk to a local vLLM server:

- to: vllm://chat?model=default&baseURL=http://localhost:8000/v1

Remote GPU host with auth:

- to: vllm://chat?model=llama-3-70b&baseURL=https://vllm.internal:8000/v1&apiKey=${VLLM_TOKEN}

Pair with an embedding-only vLLM instance on a different port:

- to: vllm://embeddings?model=bge-large&baseURL=http://embed.internal:8001/v1

Behavior notes¶

served-model-name. vLLM advertises a single model name; the model URI param must match it (or be the model's HuggingFace repo name if --served-model-name was not set on the server).
300 s timeout instead of the usual 120 s — covers cold-start / KV-cache rebuild scenarios.
All other behaviour matches openai://.

Source¶

pkg/component/vllm/vllm.go.
pkg/component/llmutil/llmutil.go.

vllm://¶