Skip to content

Add an API to bind jq variables ($name) from the host, like gojq's WithVariables #35

Description

@gh123man

Operation / feature

An API to bind jq variables ($name) from the host, equivalent to gojq's
WithVariables or the jq CLI's
--arg / --argjson:

p, err := fastjq.Compile(query, fastjq.WithVariables("$api_key", "$env"))
out, err := p.Run(input, apiKeyJSON, envJSON)

fastjq supports expr as $x | body bindings inside a query, so the machinery exists —
what's missing is a way for the caller to supply the values.

Why it matters beyond convenience

The natural use case is "the query is a fixed program, the values are data" — which is
precisely the case where the values are untrusted or sensitive. Without a host binding
API, the obvious workaround is to build the query as a string:

// Please don't do this
query := fmt.Sprintf(`%s as {$api_key} | %s`, argsJSON, transform)

That has two problems, and I think they're worth calling out in the docs regardless of
whether this feature lands:

  1. It's an injection surface. json.Marshal happens to make it safe (it escapes " and
    \, which neutralises both string break-out and jq's "\(...)" interpolation), but
    that's a non-obvious property to be relying on, and hand-built JSON would not be safe.
  2. It leaks secrets into error messages. Once a value is in the query text, it can
    surface in any parse error — e.g. unexpected trailing input: %q at query.go:295 echoes
    remaining query text. If that value is a credential, it lands in the caller's logs.

The workaround I settled on avoids both by passing values through the input instead of the
query, wrapping the document and destructuring in a generated prelude:

// input:  {"config": <real input>, "arguments": {"api_key": "..."}}
// query:  .arguments as {$api_key} | .config | <transform>

That works and keeps values out of compile errors, but it forces the caller to reserve key
names in the input document, to hand-validate that every argument name is a jq identifier,
and to reject names in jq's built-in variable namespace (an argument named __loc__
otherwise silently resolves to $__loc__'s source-location object rather than the supplied
value). All of that is boilerplate a WithVariables equivalent would remove.

Values still appear in runtime errors under any scheme, since jq names the offending
value — that's inherent, and callers should scrub. But compile-time leakage is avoidable and
a host binding API avoids it by construction.

Reference

Expected allocation tier

  • Zero allocations on the hot path (must be implementable on raw bytes
    without buffering structured data; see docs/CONSTRAINTS.md).

Values would arrive as []byte JSON, same as the input, so variable lookup is a slice
reference — execContext.lookupVar already returns ([]byte, bool). Binding N variables at
program start should be O(N) with no per-record cost.

Context

Found while replacing gojq with fastjq in the Datadog Agent. The call site applies a
remote-config-authored jq transform to a config file, with credentials supplied separately as
named arguments specifically so the transform can stay a static, reviewable program.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions