Operation / feature
An API to bind jq variables ($name) from the host, equivalent to gojq's
WithVariables or the jq CLI's
--arg / --argjson:
p, err := fastjq.Compile(query, fastjq.WithVariables("$api_key", "$env"))
out, err := p.Run(input, apiKeyJSON, envJSON)
fastjq supports expr as $x | body bindings inside a query, so the machinery exists —
what's missing is a way for the caller to supply the values.
Why it matters beyond convenience
The natural use case is "the query is a fixed program, the values are data" — which is
precisely the case where the values are untrusted or sensitive. Without a host binding
API, the obvious workaround is to build the query as a string:
// Please don't do this
query := fmt.Sprintf(`%s as {$api_key} | %s`, argsJSON, transform)
That has two problems, and I think they're worth calling out in the docs regardless of
whether this feature lands:
- It's an injection surface.
json.Marshal happens to make it safe (it escapes " and
\, which neutralises both string break-out and jq's "\(...)" interpolation), but
that's a non-obvious property to be relying on, and hand-built JSON would not be safe.
- It leaks secrets into error messages. Once a value is in the query text, it can
surface in any parse error — e.g. unexpected trailing input: %q at query.go:295 echoes
remaining query text. If that value is a credential, it lands in the caller's logs.
The workaround I settled on avoids both by passing values through the input instead of the
query, wrapping the document and destructuring in a generated prelude:
// input: {"config": <real input>, "arguments": {"api_key": "..."}}
// query: .arguments as {$api_key} | .config | <transform>
That works and keeps values out of compile errors, but it forces the caller to reserve key
names in the input document, to hand-validate that every argument name is a jq identifier,
and to reject names in jq's built-in variable namespace (an argument named __loc__
otherwise silently resolves to $__loc__'s source-location object rather than the supplied
value). All of that is boilerplate a WithVariables equivalent would remove.
Values still appear in runtime errors under any scheme, since jq names the offending
value — that's inherent, and callers should scrub. But compile-time leakage is avoidable and
a host binding API avoids it by construction.
Reference
Expected allocation tier
Values would arrive as []byte JSON, same as the input, so variable lookup is a slice
reference — execContext.lookupVar already returns ([]byte, bool). Binding N variables at
program start should be O(N) with no per-record cost.
Context
Found while replacing gojq with fastjq in the Datadog Agent. The call site applies a
remote-config-authored jq transform to a config file, with credentials supplied separately as
named arguments specifically so the transform can stay a static, reviewable program.
Operation / feature
An API to bind jq variables (
$name) from the host, equivalent to gojq'sWithVariablesor the jq CLI's--arg/--argjson:fastjq supports
expr as $x | bodybindings inside a query, so the machinery exists —what's missing is a way for the caller to supply the values.
Why it matters beyond convenience
The natural use case is "the query is a fixed program, the values are data" — which is
precisely the case where the values are untrusted or sensitive. Without a host binding
API, the obvious workaround is to build the query as a string:
That has two problems, and I think they're worth calling out in the docs regardless of
whether this feature lands:
json.Marshalhappens to make it safe (it escapes"and\, which neutralises both string break-out and jq's"\(...)"interpolation), butthat's a non-obvious property to be relying on, and hand-built JSON would not be safe.
surface in any parse error — e.g.
unexpected trailing input: %qat query.go:295 echoesremaining query text. If that value is a credential, it lands in the caller's logs.
The workaround I settled on avoids both by passing values through the input instead of the
query, wrapping the document and destructuring in a generated prelude:
That works and keeps values out of compile errors, but it forces the caller to reserve key
names in the input document, to hand-validate that every argument name is a jq identifier,
and to reject names in jq's built-in variable namespace (an argument named
__loc__otherwise silently resolves to
$__loc__'s source-location object rather than the suppliedvalue). All of that is boilerplate a
WithVariablesequivalent would remove.Values still appear in runtime errors under any scheme, since jq names the offending
value — that's inherent, and callers should scrub. But compile-time leakage is avoidable and
a host binding API avoids it by construction.
Reference
--arg/--argjsonWithVariables— namesare declared at compile time, values passed positionally to
RunExpected allocation tier
without buffering structured data; see
docs/CONSTRAINTS.md).Values would arrive as
[]byteJSON, same as the input, so variable lookup is a slicereference —
execContext.lookupVaralready returns([]byte, bool). Binding N variables atprogram start should be O(N) with no per-record cost.
Context
Found while replacing gojq with fastjq in the Datadog Agent. The call site applies a
remote-config-authored jq transform to a config file, with credentials supplied separately as
named arguments specifically so the transform can stay a static, reviewable program.