vlm://¶

Provider-portable vision facade. One URI shape that routes to openai://, anthropic://, ollama://, or vertex:// under the hood — so you can swap providers via a single param without re-writing your route. Also handles per-page PDF fan-out so multi-page documents don't need a separate file:// + split pipeline.

vlm:// does not re-implement vision. It composes the existing LLM components, so every fix and feature in those components flows through. Use it when you want one consistent vision URI; use the underlying components directly when you want provider-specific features (Anthropic prompt caching, OpenAI structured outputs, etc.).

URI shape¶

vlm://?provider=<openai|anthropic|ollama|vertex>&model=…&apiKey=…

When to use what¶

Parameters¶

Provider selection¶

Generation¶

PDF page-splitting¶

PDF→image rendering¶

vlm:// handles PDF rendering automatically when the chosen provider needs it. The renderer=auto default picks the right path per provider so you don't have to know which providers accept PDF and which don't. The route is the same regardless of provider.

Renderer modes:

Native renderer details:

• Built on klippa-app/go-pdfium running Google's PDFium engine inside the Wazero pure-Go WASM runtime.
• License chain: MIT (wrapper) + Apache 2.0 (PDFium + Wazero). No CGO, no AGPL, no commercial licensing fees.
• First call pays ~3 s for WASM module compile; subsequent renders are sub-millisecond.
• Pool is process-wide, reused across all vlm:// calls.

Headers¶

vlm:// uses the same multimodal header surface as the underlying LLM components, so a header set on a vlm:// exchange flows straight through.

Examples¶

1. Caption a single image — swap providers via one param¶

- id: caption-via-openai
  from: direct://caption
  steps:
    - to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}&prompt=Caption%20this%20image

Same route, different provider:

- to: vlm://?provider=anthropic&model=claude-3-5-sonnet-latest&apiKey=${env:ANTHROPIC_API_KEY}&prompt=Caption%20this%20image

- to: vlm://?provider=ollama&model=granite3.2-vision&baseURL=http://localhost:11434&prompt=Caption%20this%20image

2. PDF page-splitting + stitching — same route, any provider¶

The renderer=auto default makes this work uniformly:

# Anthropic — PDFs sent as-is, no rendering
- to: vlm://?provider=anthropic&model=claude-3-5-sonnet-latest&apiKey=${env:ANTHROPIC_API_KEY}&splitPDF=true&concurrency=4&prompt=Convert%20this%20page

# OpenAI — vlm:// auto-renders each page to PNG via PDFium WASM, then dispatches
- to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}&splitPDF=true&concurrency=4&prompt=Convert%20this%20page

# Ollama — same auto-render path
- to: vlm://?provider=ollama&model=granite3.2-vision&baseURL=http://localhost:11434&splitPDF=true&prompt=Convert%20this%20page

# Vertex — PDF passthrough
- to: vlm://?provider=vertex&model=gemini-2.0-flash&apiKey=${env:GOOGLE_API_KEY}&splitPDF=true&prompt=Convert%20this%20page

2b. Force native rendering for cost reasons¶

# Anthropic accepts PDF, but you've measured that image tokens are
# cheaper for your particular usage. Force native rendering.
- to: vlm://?provider=anthropic&model=claude-3-5-sonnet-latest&apiKey=${env:ANTHROPIC_API_KEY}&splitPDF=true&renderer=native&renderDPI=200

2d. GPT-5 / reasoning models¶

GPT-5 and the o-series have two wire quirks vlm:// handles automatically:

1. max_tokens is renamed to max_completion_tokens when the model id starts with o1/o3/o4/gpt-5.
2. temperature is omitted from the wire — OpenAI rejects any value other than the default 1. Helix drops the field; OpenAI's default applies.

You keep writing the same URI; no per-model knowledge required:

- to: vlm://?provider=openai&model=gpt-5&apiKey=${env:OPENAI_API_KEY}
  params:
    maxTokens: 16384               # → max_completion_tokens on the wire
    reasoningEffort: high          # minimal | low | medium | high | xhigh
    verbosity: medium              # low | medium | high
    seed: 42                       # supported on GPT-5
    responseFormat: json_object    # supported on GPT-5 / 5.5
    prompt: "Extract dosages as JSON."

The Responses API (/v1/responses) is not supported yet — gpt-5-pro and gpt-5.2-pro may not reach via this chat-completions path. Use the agent:// component for those, or wait for Phase 2 of the LLM-params work.

2c. Bridge through an existing docling-serve¶

# You already run docling-serve. Avoid bundling PDFium for this route
# and reuse the docling renderer.
- to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}&splitPDF=true&renderer=docling&renderEndpoint=http://docling:5001

3. Per-request overrides via headers¶

- id: dynamic-prompt
  from: direct://caption-with-context
  steps:
    - set-header:
        name: HelixLLMSystemPrompt
        starlang: |
          "Describe this " ++ headers["doc_type"] ++ " image. Skip decoration."
    - set-header:
        name: HelixLLMImageDetail
        constant: "high"
    - to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}
      # URI prompt would be overridden by HelixLLMSystemPrompt above; not set here.

4. Mixed text + image URL¶

- set-body: "Is this medication safe with metformin? Use the package image to answer."
- set-header:
    name: HelixLLMImages
    starlang: |
      [ "https://internal.example.com/pills/" ++ payload.med_id ++ ".jpg" ]
- to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}

5. Failover by route choice¶

vlm:// itself doesn't do failover (planned for a follow-up round). For now use a choice: step with an error handler:

- to: vlm://?provider=openai&model=gpt-4o&apiKey=${env:OPENAI_API_KEY}
  on-exception:
    - to: vlm://?provider=anthropic&model=claude-3-5-sonnet-latest&apiKey=${env:ANTHROPIC_API_KEY}

Behaviour notes¶

• Composes, does not re-implement. A vlm://?provider=openai call lands in the exact same code path as a direct openai://chat call — same HTTP client, same auth, same parsing. So multimodal fixes shipped on the underlying components flow through automatically.
• Body shape conventions follow the underlying LLM components: []byte with image magic → image; string / non-image bytes → text; *exchange.MultipartContent → structured pass-through; the HelixLLMImages header appends additional images.
• No streaming yet. All providers run in non-streaming mode here. If you need streaming, use the direct component or the agent:// layer.
• No tools / function calling. Same caveat as the direct LLM components — use agent:// for those features.
• Concurrency is bounded but not rate-limited. Set concurrency to match your provider tier. Excess parallelism leads to 429s, which surface as the first error in the page set.
• Stitched output is text only. Per-page responses are concatenated by pageSeparator. If you need per-page structured metadata, use the direct component in a custom split/aggregate pipeline.

Source¶

• pkg/component/vlm/vlm.go — Component, Endpoint, Describe
• pkg/component/vlm/params.go — URI parser + page-range
• pkg/component/vlm/producer.go — dispatch + PDF split/fan-out
• pkg/component/vlm/headers.go — HelixVLM* response headers
• pkg/component/vlm/vlm_test.go — 14 httptest-stubbed cases

See also¶

• openai, anthropic, ollama, vertex — underlying providers.
• docling — parser-style PDF extraction with deterministic table structure.
• agent — full LLM surface (tools, memory, streaming).