Repository navigation
mergefix2 - #209
Merged
Merged
mergefix2#209
Conversation
CI only builds on macOS with libc++, which transitively provides several standard headers that libstdc++ does not, and links against a monolithic LLVM dylib that resolves every component. Neither holds on Linux with GCC and a component-wise LLVM install, so the build failed outright. - Add the standard headers that were being picked up transitively: <cmath> (CPP.cpp), <cstdint> and <algorithm> (Token.h), <functional> (Trees.cpp, Layouts.cpp, Defers.cpp), <unordered_map> (DCE.cpp, LoopTransforms.cpp). - Request the LLVM components src/CodeGen uses directly: mc, targetparser, codegen, transformutils, instcombine, scalaropts, ipo.
Brings over the IR-level machinery the paper's associative and idempotent
reductions need, without any of the benchmarking scaffolding:
- AggOp{avg, count, prod, sum}, the set-level reductions of Fig. 2 that
take no lambda.
- SetOp::minimum, the paper's set-level `min`. Spelled `minimum` because
`min` is already the binary scalar intrinsic and the vector reduction.
- Annotation::Aggregate, the reduction augmentations of Section 4
(`with min(id) = idl`), alongside Data/Volume/Interval.
- Accumulate::Min/Max, plus LLVM and CUDA emission for them and for the
previously unhandled Argmax.
- Scan carries an optional reduction op, so a scan can aggregate subtree
results with something other than set union.
- Infinity becomes Extrema{inf, eps}; reductions need an epsilon for
bounds that are not tight. `eps` is now parseable alongside `inf`, and
both still print lowercase so they round-trip with the surface syntax.
Deliberately not ported from that branch: the commented-out else-if chain
in Printer::visit(IfElse), and the machine_epsilon change. Both are
unrelated to operators and would churn every golden.
- `minimum(metric, set)` and the set aggregations `avg` / `count`. - `sum` and `prod` share a spelling with the vector reductions, so they dispatch on the argument's type in VectorReduce::make rather than getting their own parser entry. - Reduction augmentations in the schedule: `with min(id) = idl`. - `eps[[T]]` now produces a typed Extrema instead of being folded to a float immediate at parse time. Adding `count` as a builtin collided with layout field names, e.g. the `count : u32; group[count] index : u32` in lower/trace.bonsai. A builtin name now only denotes an intrinsic where it is actually applied, as `f(x)` or `f[[T]](x)`. A bare builtin name was already an error, so this widens what parses rather than narrowing it.
The Rewriter that walks a Match now also collects, per arm, the reduction augmentations a node stores and any volumes annotated on its children, alongside the bounding volume and scalar intervals it already tracked. build_minimum implements Algorithm 2: value-based pruning comes from fusing `M(x) < best` into the inner filter and letting predicate analysis bound it, with the accumulator's interval recorded as (-inf, best]. An empty-able set yields an option, matching build_argmin. Predicate analysis gains the rules that reductions over scalar fields need: Div, Select, Access, and Intrinsic::max, plus tighter Mul bounds. A geometric operand that varies without a bounding volume now yields an undefined interval instead of an internal error -- "cannot be bounded" is an ordinary outcome that makes the traversal recurse unconditionally. For the same reason, a filter whose upper bound is trivially true no longer emits a guard. count is lowered here too, but as the branch's special case; the next commit replaces it with the general map-then-reduce form.
The ported count lowering was a special case: a bespoke accumulator that only fired when a node stored a `count()` augmentation. Replace it with the general form the paper describes, where count is a map of every element to 1 followed by an add-reduction with identity 0. - AggOp gains `reduce`, carrying an identity and an associative, commutative binary combiner. `count`, `sum` and `prod` become sugar that expand_aggregate() rewrites into it. - build_map implements Algorithm 1 lines 9-12, and Scan now carries the map applied to subtree elements so a scan under a map stays faithful. - build_reduce implements Algorithm 1 lines 13-22. It recognises which augmentation, if any, computes the reduction over a subtree by matching the combiner against the map feeding it: summing a constant 1 is a `count()`, summing a field f is a `sum(f)`. When a node stores that aggregate the subtree is folded in wholesale; otherwise the scan carries the reduction and recurses. - Updates are emitted as an Accumulate for combiners the backends can apply in place, and otherwise as a store of the inlined combiner, so an arbitrary user-written reduce still lowers. `avg` still errors: it needs a pair reduction over a running sum and count, which the combiner machinery does not build yet.
Adds maximum, argmax, any and all, and general reduce, so every operator in the paper's Figure 2 now lowers onto a tree. - SetOp::make is restructured around shared validation (the lambda must accept the set's elements; the result is optional when the set can be empty), replacing the near-duplicate argmin and minimum branches. - minimum/maximum and argmin/argmax are one lowering each, parameterized by which end of the metric they seek: the identity, the accumulate op, the fused "improves on best" comparison, the accumulator's own interval and which subtree bound tightens it all follow from that direction. - build_extremum implements the rest of Algorithm 2: an included subtree is folded in wholesale when the node stores the matching min/max augmentation, and a recursive call first tightens the accumulator with the metric's bound over the subtree. - build_quantifier implements Algorithm 4. Both directions short-circuit once the accumulator reaches the absorbing element of the boolean lattice, and skip a subtree whose predicate is proven always or never true. - reduce(identity, combiner, set) is now surface syntax; `sum` over a set and an explicit reduce produce identical traversals. Two bugs fixed along the way, both about bounds not being found: predicate_analysis ignored a tagged interval on the top-level expression, because it accepted the expression directly instead of going through the interval-map lookup that its subexpressions use; and build_extremum built its volume map without the matching interval map. Together these made `minimum(|p| p.id, points)` tighten the accumulator with a dangling `p.id` instead of the node's stored upper bound.
Extends GeomOp with the rest of Figure 1 -- covers, disjoint, equals, touches, within, and the per-dimension ordering predicates lex/ley/lez and ltx/lty/ltz -- and replaces the three hand-written cases in predicate analysis with the paper's two tables. geom_upper_bound is Algorithm 5 and geom_lower_bound Algorithm 6 lines 1-13, each transcribed case by case over the three configurations the paper enumerates: uniform-varying, varying-uniform, and both varying. An undefined result is the algorithms' trailing `_ |-> true` and `_ |-> false`. Metric bounds come from Algorithm 6 lines 15-25, which also fills in distmax; only distmin was handled before. The ordering predicates are the one family the paper leaves out "for brevity". They are monotonic, so the sufficient condition is the same relation between the bounding volumes, and the necessary condition is the negation of the strict counterpart with the operands swapped. Two deliberate deviations, both noted at the rules: - contains(v, u) keeps this repo's contains(V_v, u) upper bound rather than the paper's intersects(V_v, u). It is strictly tighter and equally sound: a b inside a is inside a's volume too. - equals(v, u) uses within(u, V_v), matching the symmetric case, since equality constrains the uniform operand to fit inside the varying operand's volume regardless of argument order. lower/bvh.expect changes because filters on `intersects` now get a sufficient condition. That is the specialization the paper's Figure 5b shows, where `always` becomes `contains`.
stdlib/topology.bonsai defines the topological and ordering relations for the pairs of shapes they are meaningful on. The AABB-against-AABB relations are not optional: predicate analysis derives its pruning conditions in terms of bounding volumes, so it emits calls like covers(u, V) and disjoint(V0, V1) that lower-geometrics has to resolve. The rest are what user queries are written against: AABB against Point, Sphere and Triangle, with box-triangle overlap done by the separating axis test. A ray has empty interior in three dimensions, so it can only contain degenerate geometry; contains(Ray, ...) reports false. That is sound in both roles the rule plays -- as a sufficient condition a false negative only costs pruning, and under negation it is the weakest necessary condition. stdlib/distance.bonsai gains the metrics on bounding volumes that Algorithm 6 lines 15-25 need: every distmin a query uses has to have a distmax counterpart on the volumes, and a closest-point query over a Point tree needs distmax(Point, AABB), which was missing. Verified by lowering one query per predicate against these definitions and checking each derived always/maybe pair against Algorithms 5 and 6. The two trace goldens grow because they import stdlib/distance and print the whole program.
Section 6.2 gives rules for and, or and not, but only `&&` was implemented. Predicates are usually written with `&` and `|`, which on booleans mean the same thing, and negation appeared only as something the analysis itself produced (disjoint's upper bound) and never as something it could analyse. - `&`, `|` and `||` join `&&`: all four are monotonically increasing in both arguments, so each bound is that operator over the corresponding bounds. - Negation replaces a hard error. Logical not is monotonically decreasing, so its bounds are the negated bounds swapped; arithmetic negation reverses the order of the bounds the same way. With these, the paper's Section 2 range query `filter(|x| lo <= x.id & x.id <= hi, ...)` lowers to exactly the always/maybe pair the paper shows.
The min/max augmentation lookup added with build_extremum could never fire. Fusing the value-based condition into a filter is what makes predicate analysis prune on it, but with no filter to fuse into the wrapper was synthesised anyway, and its always-branch -- the `scan` construct standing for a wholly included subtree -- was then dropped because `M(x) < best` has no lower bound. The inclusion case of Section 2 was unreachable. Now the wrapper is only built when there is a real filter to fuse into. Lowering the set as it is keeps the scan, and build_extremum handles it: a node storing this extremum over its subtree settles the contribution exactly and is never visited, and otherwise the subtree is skipped when it cannot beat the running best and tightens the accumulator when it can. `minimum(|p| p.id, points)` over a tree annotated `with min(id) = idl` now reads the root's stored minimum and stops, and without the annotation falls back to the `min_wmax` traversal on page 10.
tests/bonsai/lower/setops covers each operator of Figure 2 over one small interval tree, so the goldens show the traversal each lowering produces and nothing else. The range predicate is the paper's Section 2 example, which makes the always and maybe conditions directly comparable to it. Separate augmented variants show a stored count or extremum folding a whole subtree in without visiting it. tests/bonsai/lower/geomops covers each geometric operator, in both operand orders where Algorithms 5 and 6 give different rules, plus a dual-tree query for the both-varying cases and a metrics query for Algorithm 6 lines 15-25. These stop before lower-geometrics, so they check the derived conditions without needing geometry implementations. stdlib-resolution.bonsai goes all the way through against the standard library and so checks the other half: that every operator the analysis emits on a bounding volume has an implementation to resolve to. Also fixes runtime/bonsai_cpp.h, which pulled std::round, std::abs and friends into the global namespace and used bit_cast and memcpy without including the headers that declare them. libc++ supplies them transitively, so this only showed up off macOS -- it broke every correctness/cpp test on Linux. Three of the four now pass; vec still fails on what looks like a vector-ABI difference between the generated object code and the hand-written driver.
correctness/cpp/setops builds a real interval tree and checks count, sum, minimum, maximum, any and all against an unpruned scan over the same data, across ranges that are empty, inside one subtree, spanning several, and covering everything. A traversal that prunes a subtree it should have visited, or double-counts one it folded in from an augmentation, shows up as a mismatch rather than as a golden that merely looks plausible. Getting there needed the parts of the pipeline a reduction over a tree touches, which nothing had exercised before: - A scan over a wholly included subtree can now carry a reduction and a map, so it folds elements into an accumulator instead of unioning them into a set. That was the standing "support scan augmentations" TODO. Scan carries the accumulator it reduces into, and scan functions are shared per (trees, reduction, map). - A leaf's array is iterated elementwise when reducing. Appending an iterable adds all of it at once, which is right for a set and wrong for an accumulator. - The C++ backend gained Accumulate, scalar Allocate, Extrema and Build, and now passes mutating arguments by reference -- without which a recursive traversal updated a copy of the accumulator and the result was always the identity. Also drops tests/bonsai/error/string-literal.expect, which has no corresponding .bonsai and is not registered, and corrects the label list in tests/README.md.
CI runs clang-format 19 over the diff.
Algorithm 5 line 17 gives the upper bound of equals(v, u) as contains(V_v, u), not within(V_v, u) as I first read it. The two differ only in which operator names the relation, and contains(V_v, u) is exactly the within(u, V_v) that was already being emitted -- so the behaviour was right, but the earlier commit described a departure from the paper that does not exist. Use the paper's spelling. Each single-varying case now keeps the volume in the position its operand held, which is the convention the algorithms follow throughout: within(u, V_v) when the second operand varies, contains(V_v, u) when the first does. The contains(v, u) deviation noted earlier does stand: Section 6.3 states that bound as intersects(V_v, u), where this repo uses the tighter and equally sound contains(V_v, u).
Checked every rule in Algorithms 5 and 6 against the text extracted from the paper rather than against a rendering of the page, which is how the equals rule got misread. Two upper bounds were wrong: - covers(v, u) is covers(V_v, u) on line 9, not intersects(V_v, u). Everything a varying operand covers lies inside its bounding volume, so this prunes strictly more. - disjoint(v, u) is not-within(V_v, u) on line 12. Same relation as the not-contains(u, V_v) being emitted, but spelled with the volume in the position its operand held, which is the convention the algorithms follow and which decides the overload lower-geometrics resolves to. Everything else matches, including all of Algorithm 6 and the metric bounds. The one remaining deviation is contains(v, u), where line 6 gives intersects(V_v, u) and this repo uses the tighter contains(V_v, u); the comment there now notes the paper uses that same tight form for the corresponding covers case.
Two findings from replaying the pldi-benchmarks suite against this branch. The closest-point benchmark lost a pruning step. Synthesizing the `M(x) < best` filter does two things: it lets predicate analysis prune on the value, and it puts the node's distance bound in front of every match arm, so a leaf popped from the traversal queue is rejected before its primitives are scanned. Skipping that whenever there was no filter to fuse into gave up the second for subtrees that could never have used a stored extremum anyway. Now it is skipped only when the metric names a field an augmentation might cover, which is exactly when the `scan` it consumes is worth keeping. Counting a leaf became a loop. The elements of a leaf are summed one at a time, where asking the leaf how many it holds is a single operation. Summing the same constant over every element is now that constant times the count, which restores the generated code the benchmarks had. This needs the free `count()` overloads on `set` and `range`, ported from pldi-benchmarks; correctness/cpp/setops covers the new path.
apps/pbrt/trace.bonsai and both rtiow variants declared a leaf holding an array of primitives but never tagged it as the data the tree stores, so lowering built a match arm that neither yields nor recurses and tripped an assertion. Every other tree in the repository carries the tag, e.g. `Leaf(d : array[f32]) with data = d` in apps/queries/range. rtiow additionally declared `tree[[Sphere]]` while its set, its leaf and its layout all hold MaterialSphere. The primitive type is what the tree stores; the bounding volume stays a Sphere, which is what the `with Sphere(center, radius)` annotation is for. All three now compile: pbrt and rtiow/cpu to a closest-hit traversal guarded by intersects and distmin on the node volume, and rtiow/cuda through the CUDA backend. These were reached earlier only after this branch's parser fix let `group[count]` through; before that they failed at parse time, which is why the underlying problem had not surfaced.
Building apps/queries/rt required CGAL even though CGAL is only one of its three baselines, alongside FCPW and Bonsai. That is a heavy dependency to impose on anyone who wants to reproduce the FCPW comparison, which is the one the paper's Figure 7b reports. Building with -DBONSAI_NO_CGAL now drops the CGAL headers, its types and reference intersector, run_cgal, and the call site, so the benchmark compiles with no CGAL on the include path at all and runs FCPW against Bonsai. Without the define nothing changes. Note that the tree copy sitting between the CGAL types and run_cgal is inside a block comment, so it is not part of the guarded region.
The benchmark segfaulted on dragon.obj and hairball.obj. Both are Blender exports that use negative, relative face indices -- `f -435545//-438368` -- and the OBJ reader treated every index as absolute, so `atoi(idx) - 1` became a huge unsigned offset and construct_triangle walked off the vertex array. Negative indices now count back from the end of the vertex list, as the format specifies. Two neighbouring problems in the same loop, from the same source: the token array could overrun on a face with more than 30 vertices, and fanning a polygon into triangles read the wrong vertices for the second and later triangles. Separately, the FCL install path was hardcoded to an absolute path on one machine, so nobody else could configure the benchmark. It keeps that default and is now overridable with -DFCL_PATH=. All four scene pairs run against FCL with these. The hairball against rotated-hairball pair reports one more collision than FCL does, 5118441 versus 5118440; that is unchanged from before this branch, and 5118441 is the count the paper reports.
The compiler on main is the general one; the branches carrying the paper's benchmarks and artifact are easy to miss from here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.