Operation / feature
A context.Context-aware run API, equivalent to gojq's
RunWithContext:
func (p *Program) RunFuncContext(ctx context.Context, input []byte, fn func([]byte) error) error
func (p *Program) RunContext(ctx context.Context, input []byte) ([]byte, error)
Why
There is currently no way to abort a running program. A query whose output is unbounded
runs to completion no matter what the caller wants:
range(1000000000) | {value: .}
RunAll additionally materialises every output before returning, so that query is an
unbounded memory commitment as well as an unbounded CPU one. RunFunc at least lets the
caller stop accumulating — though today the callback can't actually halt the walk either
(#31), so even a caller who counts outputs keeps getting invoked.
For a library aimed at high-throughput log processing this also matters for ordinary
operational reasons: per-record deadlines, shutdown, and load-shedding all want
cancellation, not just pathological-input defence.
The workaround available today is an output-count cap inside the callback, which is what we
did. It bounds output cardinality but nothing else — it can't interrupt time spent inside a
single output, and it doesn't help queries that are slow rather than prolific.
Reference
- gojq:
Code.RunWithContext
checks ctx.Err() during iteration and yields the context error as a result.
Expected allocation tier
A ctx.Err() check between outputs (and at generator/reduce/foreach loop boundaries) is
a nil-pointer comparison in the common case and allocates nothing. Keeping the existing
non-ctx methods as the zero-overhead path and having them delegate with
context.Background() would avoid any cost for callers that don't need it.
Context
Found while replacing gojq with fastjq in the Datadog Agent. The call site executes
remote-config-authored jq transforms in-process during package configuration; the gojq
version passed the operation's context.Context straight through to RunWithContext, and
that cancellation was lost in the port.
Operation / feature
A
context.Context-aware run API, equivalent to gojq'sRunWithContext:Why
There is currently no way to abort a running program. A query whose output is unbounded
runs to completion no matter what the caller wants:
RunAlladditionally materialises every output before returning, so that query is anunbounded memory commitment as well as an unbounded CPU one.
RunFuncat least lets thecaller stop accumulating — though today the callback can't actually halt the walk either
(#31), so even a caller who counts outputs keeps getting invoked.
For a library aimed at high-throughput log processing this also matters for ordinary
operational reasons: per-record deadlines, shutdown, and load-shedding all want
cancellation, not just pathological-input defence.
The workaround available today is an output-count cap inside the callback, which is what we
did. It bounds output cardinality but nothing else — it can't interrupt time spent inside a
single output, and it doesn't help queries that are slow rather than prolific.
Reference
Code.RunWithContextchecks
ctx.Err()during iteration and yields the context error as a result.Expected allocation tier
without buffering structured data; see
docs/CONSTRAINTS.md).A
ctx.Err()check between outputs (and at generator/reduce/foreachloop boundaries) isa nil-pointer comparison in the common case and allocates nothing. Keeping the existing
non-
ctxmethods as the zero-overhead path and having them delegate withcontext.Background()would avoid any cost for callers that don't need it.Context
Found while replacing gojq with fastjq in the Datadog Agent. The call site executes
remote-config-authored jq transforms in-process during package configuration; the gojq
version passed the operation's
context.Contextstraight through toRunWithContext, andthat cancellation was lost in the port.