Recursive user-defined function causes a fatal stack overflow that recover() cannot catch

[gh123man]

Description

A recursive user-defined function overflows the goroutine stack. In Go this is a
fatal error: stack overflow, not a panic — recover() cannot catch it, so it takes
the calling process down with it.

This matters more than a normal crash for two reasons:

1. The README states "It never panics (enforced by fuzz tests)". That reads as a safety
   guarantee, and a caller who believes it will run untrusted queries without sandboxing.
   The fuzz tests appear to cover malformed input; they don't cover hostile queries.
2. There are already depth guards for other recursion paths (maxRecurseDepth,
   jsonParseDepthLimit), so the intent to bound recursion seems established — user-defined
   function calls just aren't covered.

There is no mitigation available to a caller. You cannot recover() it, and you cannot
inspect a compiled *Program to see whether it recurses, so the only defence would be to
run every query in a separate process.

For context on why we care: we hit this while evaluating fastjq as a gojq replacement in
the Datadog Agent, where jq programs arrive from remote configuration and are executed
in-process. A single such program is a remote process-kill.

Reproducer

Query:

def f: f; f

Input:

{"a":1}

Expected output:

jq 1.8.1 and gojq both terminate with a normal error. jq:

$ echo '{"a":1}' | jq 'def f: f; f'
jq: error: Recursion depth exceeded

At minimum, fastjq should return an error from Run/RunAll/RunFunc.

Actual output:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x14020160490 stack=[0x14020160000, 0x14040160000]
fatal error: stack overflow

goroutine 1 gp=0x140000021c0 m=9 mp=0x14000a80008 [running]:
github.com/DataDog/fastjq.bindCallContexts(...)
	.../fastjq/exec.go:5868 +0x764
github.com/DataDog/fastjq.execCall(...)
	.../fastjq/exec.go:5841 +0xe8
github.com/DataDog/fastjq.execMulti(...)
	.../fastjq/exec.go:284 +0x4c8
github.com/DataDog/fastjq.execCall.func1(...)
	.../fastjq/exec.go:5848
github.com/DataDog/fastjq.execState.withExecContext(...)
	.../fastjq/context.go:141
[repeats until the stack limit]

Minimal Go reproducer — note that the recover() never runs:

package main

import (
	"fmt"

	"github.com/DataDog/fastjq"
)

func main() {
	defer func() {
		// Never reached: a stack overflow is a fatal error, not a panic.
		if r := recover(); r != nil {
			fmt.Println("recovered:", r)
		}
	}()

	p, err := fastjq.Compile(`def f: f; f`)
	if err != nil {
		fmt.Println("compile error (acceptable outcome):", err)
		return
	}
	out, err := p.RunAll([]byte(`{"a":1}`))
	fmt.Println("returned:", out, err) // never reached
}

Other queries that compile and look likely to behave the same way

I did not run these, since each one costs a dead process, but they all Compile
successfully and have the same shape:

def f: .|f; f
def f(x): f(x); f(.)
[recurse(.)]

Worth covering whichever guard fixes this.

Suggested fix

A call-depth counter in execCall (or on execContext) that returns a normal error past
some limit, mirroring the existing maxRecurseDepth behaviour. A fuzz seed or unit test per
recursion shape above would keep it fixed.

Separately, I'd suggest narrowing the README's "never panics" wording to what the fuzz
tests actually establish (e.g. "never panics on malformed input"), so callers don't infer a
guarantee for untrusted queries.

Environment

• fastjq version: c3336f252fa23bbda9a611024f6398904330cd9a (master, untagged)
• Go version: go version go1.25.4 darwin/arm64
• OS / arch: macOS, arm64

[arcusbuilds]

Happy to take this one if nobody's on it. Most of the machinery looks like it's already there: execContext.depth gets incremented in bindCallContexts but nothing ever reads it back, so the fix is really just the missing comparison.

Before I write it, a few things I'd rather have your call on:

1. Is reusing maxRecurseDepth (10000) fine for the limit? Each level costs about 1.3 KB of stack, so that works out to roughly 13 MB. Happy to go lower if you'd prefer.
2. Do you want a new errCallDepthLimit sentinel next to errRecurseDepthLimit, or would you rather leave the error shape alone until Export typed errors so callers can distinguish syntax, runtime, and query-raised errors #37 is settled?
3. Should the README "never panics" wording and some query-side fuzz seeds go in the same PR, or stay separate?

One correction to your list while I'm here: [recurse(.)] already returns an error via errRecurseCycle, so that one's fine as is.

[gh123man]

Hey!

Is reusing maxRecurseDepth (10000) fine for the limit? Each level costs about 1.3 KB of stack, so that works out to roughly 13 MB. Happy to go lower if you'd prefer.

Sounds reasonable to me. We can always change it later if needed

Do you want a new errCallDepthLimit sentinel next to errRecurseDepthLimit, or would you rather leave the error shape alone until #37 is settled?

I think this can wait until #37 lands.

Should the README "never panics" wording and some query-side fuzz seeds go in the same PR, or stay separate?

I can make a call on this if you include it in the PR.