Skip to content

No way to distinguish empty from null output, and no lazy/first-output API #32

Description

@gh123man

Operation / feature

Two related gaps in expressing output cardinality through the current API:

1. Run cannot distinguish "no output" from "one null output". Both return
("null", nil):

package main

import (
	"fmt"

	"github.com/DataDog/fastjq"
)

func main() {
	for _, q := range []string{`empty`, `null`, `.missing`} {
		p, _ := fastjq.Compile(q)
		out, err := p.Run([]byte(`{}`))
		fmt.Printf("%-10s out=%q err=%v\n", q, string(out), err)
	}
}
empty      out="null" err=<nil>
null       out="null" err=<nil>
.missing   out="null" err=<nil>

In jq these are different: empty produces zero outputs, null and .missing each
produce one. A caller that needs to tell "the filter matched nothing" from "the filter
matched and the value is null" — a boolean predicate, an existence check — has to abandon
Run and hand-roll RunFunc with a found flag.

2. There is no lazy / first-output API. gojq exposes code.Run(v).Next(), which
evaluates lazily, so taking the first output means later sources are never evaluated and
their errors never occur. RunFunc walks eagerly, so a later source's error surfaces even
when the first output was fine:

p, _ := fastjq.Compile(`.a[], .bad.x`)
_, err := p.RunAll([]byte(`{"a":[1,2,3],"bad":"z"}`))
// err = Cannot index string with string "x"
// gojq's .Next() returns 1 and never touches .bad.x

Combined with #31 (a callback error can't stop the walk), "give me the first output" is
currently not expressible without also swallowing genuine errors from outputs the caller
never asked for.

Proposal

Either of these would resolve both halves:

  • RunFirst(input []byte) ([]byte, bool, error) — returns the first output and whether
    there was one, stopping evaluation there. Directly serves the common "predicate" and
    "extract one field" cases, and is the minimal fix.
  • A lazy iterator (p.Iter(input) returning something with Next() ([]byte, bool) plus
    an Err()), matching gojq's shape. More work, but it makes short-circuiting composable
    and would let first(...)/limit(...) semantics fall out naturally for callers.

If neither is in scope, then at minimum Run's doc comment should say that a null return
is ambiguous and point callers at RunFunc — right now the ambiguity is silent.

Reference

  • jq manual, empty: "empty returns no results at all"
  • gojq's lazy iterator: Code.Run
    Iter.Next()

Expected allocation tier

  • Zero allocations on the hot path (must be implementable on raw bytes
    without buffering structured data; see docs/CONSTRAINTS.md).

RunFirst should be strictly cheaper than the status quo — it can stop at the first output
instead of walking everything, and needs no result slice. A RunFirstWithBuffer variant
would keep it zero-alloc for repeated calls, matching RunWithBuffer.

Context

Found while replacing gojq with fastjq in the Datadog Agent, where the two call sites are
"evaluate a boolean predicate over a config document" and "extract one scalar" — both of
which want first-output-or-nothing semantics.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions