Dataformat processors¶

pkg/processor/dataformat.go provides marshal/unmarshal pairs for the common wire formats (JSON, XML, CSV, YAML) and content-encoding processors (Base64, GZip). They are simple Message Translator EIPs and back the data-format step variants exposed via the DSL.

The factory function GetDataFormat(name) returns a marshal/unmarshal pair by name for use in custom code.

Processors in this file¶

JSON: JSONMarshal / JSONUnmarshal¶

Both use encoding/json. No struct field tags are involved — marshaling reflects whatever Go value is in the body.

XML: XMLMarshal / XMLUnmarshal¶

XMLMarshal has one configurable field:

Maps marshal as nested <key>…</key> elements with deterministic (sorted) key order. Slices of maps marshal as <item> elements. Other types fall through to xml.Marshal.

XMLUnmarshal produces a map[string]any:

• Text-only elements become string.
• Element with children becomes map[string]any.
• Duplicate child tags accumulate into a []any.

CSV: CSVMarshal / CSVUnmarshal¶

Accepted body types for CSVMarshal: []map[string]any (deterministic header order via sorted keys) or [][]string (raw row matrix). Anything else returns an error.

CSVUnmarshal with Header: false returns a [][]string.

YAML: YAMLMarshal / YAMLUnmarshal¶

Uses gopkg.in/yaml.v3. Symmetric with JSON — body in is any Go value; body out is []byte for marshal, any for unmarshal. Marshal sets Content-Type: application/yaml.

Base64: Base64Encode / Base64Decode¶

Uses encoding/base64.StdEncoding. Both operate on []byte. Encode writes the encoded bytes back as the body; Decode parses the body as a base64 string and writes the decoded bytes.

GZip: GZipCompress / GZipDecompress¶

GZipCompress reads the body as bytes, compresses with compress/gzip, writes the compressed bytes back, and sets Content-Encoding: gzip.

GZipDecompress has one safety-tuning field:

Decompression reads at most MaxDecompressedSize + 1 bytes via io.LimitReader; if the read returns more, the processor errors with decompressed size exceeds limit of N bytes.

GetDataFormat¶

Factory used by the DSL reifier. Returns (marshal, unmarshal, err) for these names:

Unknown names return unknown data format: <name>.

Pitfalls¶

• XML unmarshal returns text content as plain string only when an element has no child elements; mixed-content elements become a map[string]any with the text discarded. Don't rely on the XML processor for documents with significant whitespace.
• CSV marshal of []map[string]any uses sorted keys of the first row as header. If later rows have additional keys, those values are silently dropped.
• Base64 uses standard encoding (with +/=). URL-safe encoding is not exposed via the YAML factory — call encoding/base64.URLEncoding directly in Go if needed.
• GZipDecompress returns an error rather than truncating when output exceeds MaxDecompressedSize. Tune this for legitimate large payloads.

Examples¶

Marshal a map to JSON, then to base64, then publish:

workflows:
  - id: encode-event
    trigger: { uri: direct://events }
    steps:
      - transform:
          starlang: |
            %sl 2.0
            output application/json
            input payload
            ---
            { "id": payload.id, "ts": now() }
      - process: { ref: json-marshal }
      - process: { ref: base64-encode }
      - target: kafka://events-b64

For custom code, use the factory:

marshal, unmarshal, err := processor.GetDataFormat("csv")

Source¶

pkg/processor/dataformat.go

See also¶

• transform — expression-driven body rewrites (use for non-format-specific transforms).
• convert — ConvertBody for simple type coercions like string ↔ []byte ↔ map.
• streaming_json — stream large JSON arrays without holding them all in memory.