Skip to content

mergefix2 - #209

Merged
rootjalex merged 27 commits into
ajr/pldi-benchmarksfrom
ajr/pldi26-mergefix2
Aug 30, 2026
Merged

rootjalex merged 27 commits into
ajr/pldi-benchmarksfrom
ajr/pldi26-mergefix2

Conversation

@rootjalex

Copy link
Copy Markdown
Owner

No description provided.

cgyurgyik and others added 27 commits September 10, 2025 13:31
CI only builds on macOS with libc++, which transitively provides several
standard headers that libstdc++ does not, and links against a monolithic
LLVM dylib that resolves every component. Neither holds on Linux with
GCC and a component-wise LLVM install, so the build failed outright.

- Add the standard headers that were being picked up transitively:
  <cmath> (CPP.cpp), <cstdint> and <algorithm> (Token.h), <functional>
  (Trees.cpp, Layouts.cpp, Defers.cpp), <unordered_map> (DCE.cpp,
  LoopTransforms.cpp).
- Request the LLVM components src/CodeGen uses directly: mc,
  targetparser, codegen, transformutils, instcombine, scalaropts, ipo.
Brings over the IR-level machinery the paper's associative and idempotent
reductions need, without any of the benchmarking scaffolding:

- AggOp{avg, count, prod, sum}, the set-level reductions of Fig. 2 that
  take no lambda.
- SetOp::minimum, the paper's set-level `min`. Spelled `minimum` because
  `min` is already the binary scalar intrinsic and the vector reduction.
- Annotation::Aggregate, the reduction augmentations of Section 4
  (`with min(id) = idl`), alongside Data/Volume/Interval.
- Accumulate::Min/Max, plus LLVM and CUDA emission for them and for the
  previously unhandled Argmax.
- Scan carries an optional reduction op, so a scan can aggregate subtree
  results with something other than set union.
- Infinity becomes Extrema{inf, eps}; reductions need an epsilon for
  bounds that are not tight. `eps` is now parseable alongside `inf`, and
  both still print lowercase so they round-trip with the surface syntax.

Deliberately not ported from that branch: the commented-out else-if chain
in Printer::visit(IfElse), and the machine_epsilon change. Both are
unrelated to operators and would churn every golden.
- `minimum(metric, set)` and the set aggregations `avg` / `count`.
- `sum` and `prod` share a spelling with the vector reductions, so they
  dispatch on the argument's type in VectorReduce::make rather than
  getting their own parser entry.
- Reduction augmentations in the schedule: `with min(id) = idl`.
- `eps[[T]]` now produces a typed Extrema instead of being folded to a
  float immediate at parse time.

Adding `count` as a builtin collided with layout field names, e.g. the
`count : u32; group[count] index : u32` in lower/trace.bonsai. A builtin
name now only denotes an intrinsic where it is actually applied, as `f(x)`
or `f[[T]](x)`. A bare builtin name was already an error, so this widens
what parses rather than narrowing it.
The Rewriter that walks a Match now also collects, per arm, the reduction
augmentations a node stores and any volumes annotated on its children,
alongside the bounding volume and scalar intervals it already tracked.

build_minimum implements Algorithm 2: value-based pruning comes from
fusing `M(x) < best` into the inner filter and letting predicate analysis
bound it, with the accumulator's interval recorded as (-inf, best]. An
empty-able set yields an option, matching build_argmin.

Predicate analysis gains the rules that reductions over scalar fields
need: Div, Select, Access, and Intrinsic::max, plus tighter Mul bounds. A
geometric operand that varies without a bounding volume now yields an
undefined interval instead of an internal error -- "cannot be bounded" is
an ordinary outcome that makes the traversal recurse unconditionally. For
the same reason, a filter whose upper bound is trivially true no longer
emits a guard.

count is lowered here too, but as the branch's special case; the next
commit replaces it with the general map-then-reduce form.
The ported count lowering was a special case: a bespoke accumulator that
only fired when a node stored a `count()` augmentation. Replace it with
the general form the paper describes, where count is a map of every
element to 1 followed by an add-reduction with identity 0.

- AggOp gains `reduce`, carrying an identity and an associative,
  commutative binary combiner. `count`, `sum` and `prod` become sugar
  that expand_aggregate() rewrites into it.
- build_map implements Algorithm 1 lines 9-12, and Scan now carries the
  map applied to subtree elements so a scan under a map stays faithful.
- build_reduce implements Algorithm 1 lines 13-22. It recognises which
  augmentation, if any, computes the reduction over a subtree by matching
  the combiner against the map feeding it: summing a constant 1 is a
  `count()`, summing a field f is a `sum(f)`. When a node stores that
  aggregate the subtree is folded in wholesale; otherwise the scan
  carries the reduction and recurses.
- Updates are emitted as an Accumulate for combiners the backends can
  apply in place, and otherwise as a store of the inlined combiner, so an
  arbitrary user-written reduce still lowers.

`avg` still errors: it needs a pair reduction over a running sum and
count, which the combiner machinery does not build yet.
Adds maximum, argmax, any and all, and general reduce, so every operator
in the paper's Figure 2 now lowers onto a tree.

- SetOp::make is restructured around shared validation (the lambda must
  accept the set's elements; the result is optional when the set can be
  empty), replacing the near-duplicate argmin and minimum branches.
- minimum/maximum and argmin/argmax are one lowering each, parameterized
  by which end of the metric they seek: the identity, the accumulate op,
  the fused "improves on best" comparison, the accumulator's own interval
  and which subtree bound tightens it all follow from that direction.
- build_extremum implements the rest of Algorithm 2: an included subtree
  is folded in wholesale when the node stores the matching min/max
  augmentation, and a recursive call first tightens the accumulator with
  the metric's bound over the subtree.
- build_quantifier implements Algorithm 4. Both directions short-circuit
  once the accumulator reaches the absorbing element of the boolean
  lattice, and skip a subtree whose predicate is proven always or never
  true.
- reduce(identity, combiner, set) is now surface syntax; `sum` over a set
  and an explicit reduce produce identical traversals.

Two bugs fixed along the way, both about bounds not being found:
predicate_analysis ignored a tagged interval on the top-level expression,
because it accepted the expression directly instead of going through the
interval-map lookup that its subexpressions use; and build_extremum built
its volume map without the matching interval map. Together these made
`minimum(|p| p.id, points)` tighten the accumulator with a dangling `p.id`
instead of the node's stored upper bound.
Extends GeomOp with the rest of Figure 1 -- covers, disjoint, equals,
touches, within, and the per-dimension ordering predicates lex/ley/lez
and ltx/lty/ltz -- and replaces the three hand-written cases in predicate
analysis with the paper's two tables.

geom_upper_bound is Algorithm 5 and geom_lower_bound Algorithm 6 lines
1-13, each transcribed case by case over the three configurations the
paper enumerates: uniform-varying, varying-uniform, and both varying. An
undefined result is the algorithms' trailing `_ |-> true` and `_ |->
false`. Metric bounds come from Algorithm 6 lines 15-25, which also fills
in distmax; only distmin was handled before.

The ordering predicates are the one family the paper leaves out "for
brevity". They are monotonic, so the sufficient condition is the same
relation between the bounding volumes, and the necessary condition is the
negation of the strict counterpart with the operands swapped.

Two deliberate deviations, both noted at the rules:
- contains(v, u) keeps this repo's contains(V_v, u) upper bound rather
  than the paper's intersects(V_v, u). It is strictly tighter and equally
  sound: a b inside a is inside a's volume too.
- equals(v, u) uses within(u, V_v), matching the symmetric case, since
  equality constrains the uniform operand to fit inside the varying
  operand's volume regardless of argument order.

lower/bvh.expect changes because filters on `intersects` now get a
sufficient condition. That is the specialization the paper's Figure 5b
shows, where `always` becomes `contains`.
stdlib/topology.bonsai defines the topological and ordering relations for
the pairs of shapes they are meaningful on. The AABB-against-AABB
relations are not optional: predicate analysis derives its pruning
conditions in terms of bounding volumes, so it emits calls like
covers(u, V) and disjoint(V0, V1) that lower-geometrics has to resolve.
The rest are what user queries are written against: AABB against Point,
Sphere and Triangle, with box-triangle overlap done by the separating
axis test.

A ray has empty interior in three dimensions, so it can only contain
degenerate geometry; contains(Ray, ...) reports false. That is sound in
both roles the rule plays -- as a sufficient condition a false negative
only costs pruning, and under negation it is the weakest necessary
condition.

stdlib/distance.bonsai gains the metrics on bounding volumes that
Algorithm 6 lines 15-25 need: every distmin a query uses has to have a
distmax counterpart on the volumes, and a closest-point query over a
Point tree needs distmax(Point, AABB), which was missing.

Verified by lowering one query per predicate against these definitions
and checking each derived always/maybe pair against Algorithms 5 and 6.
The two trace goldens grow because they import stdlib/distance and print
the whole program.
Section 6.2 gives rules for and, or and not, but only `&&` was
implemented. Predicates are usually written with `&` and `|`, which on
booleans mean the same thing, and negation appeared only as something the
analysis itself produced (disjoint's upper bound) and never as something
it could analyse.

- `&`, `|` and `||` join `&&`: all four are monotonically increasing in
  both arguments, so each bound is that operator over the corresponding
  bounds.
- Negation replaces a hard error. Logical not is monotonically
  decreasing, so its bounds are the negated bounds swapped; arithmetic
  negation reverses the order of the bounds the same way.

With these, the paper's Section 2 range query
`filter(|x| lo <= x.id & x.id <= hi, ...)` lowers to exactly the
always/maybe pair the paper shows.
The min/max augmentation lookup added with build_extremum could never
fire. Fusing the value-based condition into a filter is what makes
predicate analysis prune on it, but with no filter to fuse into the
wrapper was synthesised anyway, and its always-branch -- the `scan`
construct standing for a wholly included subtree -- was then dropped
because `M(x) < best` has no lower bound. The inclusion case of Section 2
was unreachable.

Now the wrapper is only built when there is a real filter to fuse into.
Lowering the set as it is keeps the scan, and build_extremum handles it:
a node storing this extremum over its subtree settles the contribution
exactly and is never visited, and otherwise the subtree is skipped when
it cannot beat the running best and tightens the accumulator when it can.

`minimum(|p| p.id, points)` over a tree annotated `with min(id) = idl`
now reads the root's stored minimum and stops, and without the annotation
falls back to the `min_wmax` traversal on page 10.
tests/bonsai/lower/setops covers each operator of Figure 2 over one small
interval tree, so the goldens show the traversal each lowering produces
and nothing else. The range predicate is the paper's Section 2 example,
which makes the always and maybe conditions directly comparable to it.
Separate augmented variants show a stored count or extremum folding a
whole subtree in without visiting it.

tests/bonsai/lower/geomops covers each geometric operator, in both
operand orders where Algorithms 5 and 6 give different rules, plus a
dual-tree query for the both-varying cases and a metrics query for
Algorithm 6 lines 15-25. These stop before lower-geometrics, so they
check the derived conditions without needing geometry implementations.
stdlib-resolution.bonsai goes all the way through against the standard
library and so checks the other half: that every operator the analysis
emits on a bounding volume has an implementation to resolve to.

Also fixes runtime/bonsai_cpp.h, which pulled std::round, std::abs and
friends into the global namespace and used bit_cast and memcpy without
including the headers that declare them. libc++ supplies them
transitively, so this only showed up off macOS -- it broke every
correctness/cpp test on Linux. Three of the four now pass; vec still
fails on what looks like a vector-ABI difference between the generated
object code and the hand-written driver.
correctness/cpp/setops builds a real interval tree and checks count, sum,
minimum, maximum, any and all against an unpruned scan over the same
data, across ranges that are empty, inside one subtree, spanning several,
and covering everything. A traversal that prunes a subtree it should have
visited, or double-counts one it folded in from an augmentation, shows up
as a mismatch rather than as a golden that merely looks plausible.

Getting there needed the parts of the pipeline a reduction over a tree
touches, which nothing had exercised before:

- A scan over a wholly included subtree can now carry a reduction and a
  map, so it folds elements into an accumulator instead of unioning them
  into a set. That was the standing "support scan augmentations" TODO.
  Scan carries the accumulator it reduces into, and scan functions are
  shared per (trees, reduction, map).
- A leaf's array is iterated elementwise when reducing. Appending an
  iterable adds all of it at once, which is right for a set and wrong for
  an accumulator.
- The C++ backend gained Accumulate, scalar Allocate, Extrema and Build,
  and now passes mutating arguments by reference -- without which a
  recursive traversal updated a copy of the accumulator and the result
  was always the identity.

Also drops tests/bonsai/error/string-literal.expect, which has no
corresponding .bonsai and is not registered, and corrects the label list
in tests/README.md.
CI runs clang-format 19 over the diff.
Algorithm 5 line 17 gives the upper bound of equals(v, u) as
contains(V_v, u), not within(V_v, u) as I first read it. The two differ
only in which operator names the relation, and contains(V_v, u) is
exactly the within(u, V_v) that was already being emitted -- so the
behaviour was right, but the earlier commit described a departure from
the paper that does not exist.

Use the paper's spelling. Each single-varying case now keeps the volume
in the position its operand held, which is the convention the algorithms
follow throughout: within(u, V_v) when the second operand varies,
contains(V_v, u) when the first does.

The contains(v, u) deviation noted earlier does stand: Section 6.3 states
that bound as intersects(V_v, u), where this repo uses the tighter and
equally sound contains(V_v, u).
Checked every rule in Algorithms 5 and 6 against the text extracted from
the paper rather than against a rendering of the page, which is how the
equals rule got misread. Two upper bounds were wrong:

- covers(v, u) is covers(V_v, u) on line 9, not intersects(V_v, u).
  Everything a varying operand covers lies inside its bounding volume, so
  this prunes strictly more.
- disjoint(v, u) is not-within(V_v, u) on line 12. Same relation as the
  not-contains(u, V_v) being emitted, but spelled with the volume in the
  position its operand held, which is the convention the algorithms
  follow and which decides the overload lower-geometrics resolves to.

Everything else matches, including all of Algorithm 6 and the metric
bounds. The one remaining deviation is contains(v, u), where line 6 gives
intersects(V_v, u) and this repo uses the tighter contains(V_v, u); the
comment there now notes the paper uses that same tight form for the
corresponding covers case.
Two findings from replaying the pldi-benchmarks suite against this branch.

The closest-point benchmark lost a pruning step. Synthesizing the
`M(x) < best` filter does two things: it lets predicate analysis prune on
the value, and it puts the node's distance bound in front of every match
arm, so a leaf popped from the traversal queue is rejected before its
primitives are scanned. Skipping that whenever there was no filter to
fuse into gave up the second for subtrees that could never have used a
stored extremum anyway. Now it is skipped only when the metric names a
field an augmentation might cover, which is exactly when the `scan` it
consumes is worth keeping.

Counting a leaf became a loop. The elements of a leaf are summed one at a
time, where asking the leaf how many it holds is a single operation.
Summing the same constant over every element is now that constant times
the count, which restores the generated code the benchmarks had. This
needs the free `count()` overloads on `set` and `range`, ported from
pldi-benchmarks; correctness/cpp/setops covers the new path.
apps/pbrt/trace.bonsai and both rtiow variants declared a leaf holding an
array of primitives but never tagged it as the data the tree stores, so
lowering built a match arm that neither yields nor recurses and tripped
an assertion. Every other tree in the repository carries the tag, e.g.
`Leaf(d : array[f32]) with data = d` in apps/queries/range.

rtiow additionally declared `tree[[Sphere]]` while its set, its leaf and
its layout all hold MaterialSphere. The primitive type is what the tree
stores; the bounding volume stays a Sphere, which is what the `with
Sphere(center, radius)` annotation is for.

All three now compile: pbrt and rtiow/cpu to a closest-hit traversal
guarded by intersects and distmin on the node volume, and rtiow/cuda
through the CUDA backend.

These were reached earlier only after this branch's parser fix let
`group[count]` through; before that they failed at parse time, which is
why the underlying problem had not surfaced.
Building apps/queries/rt required CGAL even though CGAL is only one of
its three baselines, alongside FCPW and Bonsai. That is a heavy
dependency to impose on anyone who wants to reproduce the FCPW
comparison, which is the one the paper's Figure 7b reports.

Building with -DBONSAI_NO_CGAL now drops the CGAL headers, its types and
reference intersector, run_cgal, and the call site, so the benchmark
compiles with no CGAL on the include path at all and runs FCPW against
Bonsai. Without the define nothing changes.

Note that the tree copy sitting between the CGAL types and run_cgal is
inside a block comment, so it is not part of the guarded region.
The benchmark segfaulted on dragon.obj and hairball.obj. Both are Blender
exports that use negative, relative face indices -- `f -435545//-438368`
-- and the OBJ reader treated every index as absolute, so `atoi(idx) - 1`
became a huge unsigned offset and construct_triangle walked off the
vertex array. Negative indices now count back from the end of the vertex
list, as the format specifies.

Two neighbouring problems in the same loop, from the same source: the
token array could overrun on a face with more than 30 vertices, and
fanning a polygon into triangles read the wrong vertices for the second
and later triangles.

Separately, the FCL install path was hardcoded to an absolute path on one
machine, so nobody else could configure the benchmark. It keeps that
default and is now overridable with -DFCL_PATH=.

All four scene pairs run against FCL with these. The hairball against
rotated-hairball pair reports one more collision than FCL does, 5118441
versus 5118440; that is unchanged from before this branch, and 5118441 is
the count the paper reports.
The compiler on main is the general one; the branches carrying the
paper's benchmarks and artifact are easy to miss from here.
@rootjalex
rootjalex merged commit f090a20 into ajr/pldi-benchmarks Aug 30, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants