`helix eval`¶

Validate a flow against a JSONL evaluation dataset. Each dataset row is parsed, the routes are reified, and the count of validated cases is reported. This subcommand is a stub today: it confirms that the dataset parses, the flow loads, and every workflow reifies — actual live execution is still done via helix run (the eval-harness implementation lives in pkg/testing and is exercised by helix test).

Usage¶

helix eval <file> --dataset <jsonl>

Flags¶

Flag	Default	Description
`--dataset PATH`	unset, required	JSONL file. Each line is a JSON object with `input`, optional `expected`, and optional `agent_id`.

The flow file argument is required and is the first non-flag positional.

Behavior notes¶

Requires $HELIX_LICENSE. Verified at process start.
Reads the dataset, splits on newlines, ignores empty lines, and unmarshals each line as {"input": "...", "expected": "...", "agent_id": "..."}. Invalid JSONL fails on the first line with a clear line number.
Loads the flow file via the standard loader and reifies every workflow with the default component set — failure to reify any workflow fails the eval.
Prints one line per case with (validated). Does NOT actually execute the workflow against each input today.
Closes with a message telling you to use helix run <file> to drive inputs end-to-end. The eval harness for full live execution + scoring lives behind helix test with --eval-log / --baseline / --min-metric flags.

Examples¶

Validate a dataset against a flow:

$ helix eval app.yaml --dataset eval/cases.jsonl
Evaluation: app.yaml
Dataset: eval/cases.jsonl (12 cases)
Routes: 4
---
  [1] input="What is 2+2?"  expected="4"  agent=chat-agent  (validated)
  [2] input="Summarise the doc"  expected="..."  agent=rag-agent  (validated)
  ...
---
Validated 12 eval case(s). Flow has 4 route(s), all reify OK.

A typical dataset line:

{"input": "What is 2+2?", "expected": "4", "agent_id": "chat-agent"}

Source¶

cmd/helix/main.go lines 2665-2766 (cmdEval).

helix eval¶