`helix test`¶

Run declarative test cases (YAML/JSON files matching *_test.yaml, *-test.yaml, etc.) against the workflows in the surrounding project. The test runner reifies routes in-process — no daemon needed — and supports tag filters, eval-metric gates, regression baselines, judge-vs-human calibration, and alert webhooks. This is the only Helix command that can fail your build on subtle quality regressions, not just hard errors.

Usage¶

helix test [flags] [targets...]

Targets may be test files or directories. If omitted, the current directory is searched for *_test.{yaml,yml,json} files.

Flags¶

Flag	Default	Description
`-v`, `--verbose`	off	Print test descriptions on PASS and show more context.
`--id=ID`	unset	Only run the test with the given ID.
`--tag=TAGS`	unset	Comma-separated tag whitelist (repeatable). Tests without any matching tag are skipped.
`--exclude-tag=TAGS`	unset	Comma-separated tag blacklist (repeatable). Useful for opting out of `@redis` / `@network` suites.
`--report=FORMAT`	unset	Emit a structured report (`json`, `junit`, `html` — see `pkg/testing`).
`--report-file=PATH`	stdout	Write the report to PATH instead of stdout.
`--min-pass-rate=R`	unset	Fail if the overall pass rate (`[0,1]`) is below R.
`--min-metric=NAME=R`	unset	Fail if the named eval metric is below R (repeatable).
`--baseline=PATH`	unset	Eval-summary baseline file. First run writes it; subsequent runs diff against it.
`--max-regression=D`	unset	Fail when any metric drops by more than D versus baseline. Without it, drift is reported only.
`--update-baseline`	off	Overwrite `--baseline` with the current run; skip gating.
`--eval-log=PATH`	unset	Append one JSONL record per run for trend analysis.
`--eval-label=L`	unset	Label for the eval-log record (e.g. a git SHA).
`--calibrate=PATH`	unset	Judge-vs-human label-pairs JSONL; reports agreement + Cohen's κ.
`--min-kappa=R`	unset	Fail if Cohen's κ from `--calibrate` is below R.
`--alert-webhook=URL`	unset	POST a JSON payload to URL on eval-gate breach.

Long forms (--tag value) and --tag=value are both accepted.

Behavior notes¶

Requires $HELIX_LICENSE. Verified at process start.
Auto-loads .env from the first target's directory.
Test files are recognised by the suffixes _test.yaml, _test.yml, _test.json, -test.yaml, -test.yml, -test.json.
For each test file the runner discovers the workflows the test refers to via three strategies, in order: walk up to find an app.yaml / helix.yaml / main.yaml; sibling file with the matching stem; all non-test YAML/JSON in the directory.
Connectors are discovered the same way so routes that reference connector://<name> reify in test mode.
Output is colored: green PASS, red FAIL, yellow SKIP. FAIL lines show the captured response body (truncated at 400 chars) to make CI logs debuggable on the first read.
Test results carry SkippedReifies — routes from sibling files that couldn't be reified — and the runner prints them on passing tests so a half-broken project never silently reports full green.
Eval gates evaluate AFTER all tests run. A passing test suite that trips an eval-metric threshold still exits non-zero with eval-gate: <metric> ... below minimum.
The alert webhook receives {breaches: [...], summary: {...}}; the call is best-effort (failure is logged but doesn't change the exit code).

Exit codes¶

0 — every test passed and every gate is satisfied.
non-zero — at least one test failed or a gate was breached (test failures and eval-gate breaches both return non-zero, the failure message tells you which).

Examples¶

Run every test in the current directory:

helix test

Run a single test:

helix test --id=happy-path tests/agent_test.yaml

Skip live-Redis tests in CI:

helix test --exclude-tag=redis

Gate a CI job on overall + per-metric quality:

helix test \
  --min-pass-rate=0.95 \
  --min-metric=faithfulness=0.85 \
  --min-metric=answer_relevance=0.80 \
  --baseline=eval-baseline.json \
  --max-regression=0.05 \
  --eval-log=eval-trend.jsonl --eval-label=$(git rev-parse --short HEAD)

Calibrate a judge against a human-labeled set:

helix test --calibrate=labels.jsonl --min-kappa=0.6

Source¶

cmd/helix/main.go lines 3357-3691 (cmdTest), plus the helpers parseMinRate / parseMinMetric / runTestsInDir / runTestFile / discoverWorkflowFiles / discoverViaAppYaml / discoverConnectors.

helix test¶