tests: Google FuzzTest support and chunked container PBT#30165
Open
wdberkeley wants to merge 3 commits intodevfrom
Open
tests: Google FuzzTest support and chunked container PBT#30165wdberkeley wants to merge 3 commits intodevfrom
wdberkeley wants to merge 3 commits intodevfrom
Conversation
Adds FuzzTest as a Bazel dep and wires it into the build so that tests
can use FUZZ_TEST() for coverage-guided fuzzing or plain property-based
testing under --config=fuzztest and normal bazel test respectively.
Custom main (fuzztest_main.cc): FuzzTest's stock main can't be used
because we need the Seastar reactor. The custom main initializes GTest,
then FuzzTest, then Seastar, running RUN_ALL_TESTS() inside run_sync(
seastar::async(...)). Three integration subtleties:
- Stack limit false positive: seastar::async uses makecontext/
swapcontext; the stack pointer jumps to a different region and
FuzzTest's stack monitor reads terabytes of usage, tripping the
default 128 KiB limit. Fixed by injecting --stack_limit_kb=0
before ParseAbslFlags.
- Flag parsing order: fuzztest::ParseAbslFlags() must precede
fuzztest::InitFuzzTest() so abseil flags like --fuzz and
--stack_limit_kb are parsed. Easy to miss in a custom main.
- Seastar argv filtering: FuzzTest's abseil flags (--fuzz, etc.)
survive InitFuzzTest and must be stripped before Seastar's start()
which uses boost::program_options and rejects unknown options.
Custom bazelrc config: We define build:fuzztest directly rather than
importing the generated fuzztest.bazelrc (checked in for reference).
The generated config applies coverage instrumentation globally via
--copt=-fsanitize-coverage=..., but FuzzTest's runtime only tracks one
counter region — whichever TU registers first wins. With global
instrumentation external deps register first and shadow all Redpanda
code (0 edges covered). Our config uses per_file_copt on src/v/ only,
split into three lines because commas before @ are pattern separators
in Bazel's per_file_copt syntax. Two Seastar-specific flags are also
required: --@seastar//:system_allocator=True (segfaults under ASan with
custom allocator) and --linkopt=-fsanitize-link-c++-runtime.
redpanda_cc_fuzztest macro: delegates to _redpanda_cc_unit_test
identically to redpanda_cc_gtest; tests must dep on
//src/v/test_utils:fuzztest instead of :gtest.
…ked_hash_map Uses Google FuzzTest in PBT mode (plain bazel test; --config=fuzztest enables continuous coverage-guided fuzzing). Each test states its property explicitly in a comment. chunked_vector (ChunkedVectorPBT): VectorModelOracle: drives both chunked_vector<int32_t> and std::vector<int32_t> through the same arbitrary sequence of push_back, pop_back, pop_back_n, erase_to_end, reserve, sort, and clear. After every operation, asserts element-wise equality AND structural invariants (size == sum of fragment sizes, capacity == sum of fragment capacities, all fragments except the last are completely full). The structural check catches bugs that leave the container logically correct but physically inconsistent. SortedBinarySearch: after sorting a chunked_vector the same way as a std::vector, std::lower_bound and std::upper_bound must return the same distance-from-begin on both. Exercises the full random-access iterator contract (operator+, operator-, operator[], operator<=>); such bugs would pass simple forward-iteration tests. IndexedAccessMatchesIterator: operator[](i) and *(begin()+i) must return the same element for every valid index. Both paths compute the same fragment-index / offset math independently, so a miscalculation in either shows up here. ReverseIterationMatchesReverse: rbegin()/rend() must yield elements in the reverse of forward order, exercising the reverse_iterator adapter wrapping the random-access iterator. chunked_hash_map (ChunkedHashMapPBT): MapModelOracle: drives both chunked_hash_map<int32_t,int32_t> and std::unordered_map through the same insert_or_assign / erase / find sequence. After every operation, asserts size equality and that every key in the oracle is present in the map with the same value. InsertBatchThenIterateAll: inserts a batch of pairs (last write wins) and checks that iterating over the map visits exactly the same sorted key set as the deduplicated oracle. Targets the growth path where new chunked_vector fragments are added to the bucket and value storage.
Adds --config=fuzztest-cov which layers -fprofile-instr-generate and
-fcoverage-mapping on top of --config=fuzztest to collect LLVM source-
level coverage data during a fuzz run.
To generate an HTML coverage report:
# 1. Run the fuzzer (Ctrl-C after however long you want)
LLVM_PROFILE_FILE=/tmp/fuzz.profraw \
bazel run --config=fuzztest-cov //src/v/container/tests:chunked_pbt \
-- --fuzz=ChunkedVectorPBT.VectorModelOracle
# 2. Locate the LLVM tools from the Bazel-managed toolchain
LLVM=$(bazel info output_base)/external/toolchains_llvm++llvm+current_llvm_toolchain/bin
# 3. Merge the raw profile
$LLVM/llvm-profdata merge /tmp/fuzz.profraw -o /tmp/fuzz.profdata
# 4. Generate the HTML report
#
# Bazel compiles some files with absolute sandbox paths that no
# longer exist after the build. Use -ignore-filename-regex to skip
# those so llvm-cov doesn't choke on missing files. Our src/v/ files
# use workspace-relative paths and are readable fine.
#
rm -rf /tmp/cov-report
$LLVM/llvm-cov show \
bazel-bin/src/v/container/tests/chunked_pbt \
-instr-profile=/tmp/fuzz.profdata \
-ignore-filename-regex='\.cache/bazel|/external/|^base64\.c' \
-format=html \
-output-dir=/tmp/cov-report
# 5. Serve and open in a browser
python3 -m http.server 8080 --directory /tmp/cov-report
# then open http://<host>:8080
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces Google FuzzTest support in the Bazel build and adds property-based tests (PBTs) for chunked_vector and chunked_hash_map that can also be run as coverage-guided fuzz tests.
Changes:
- Add a FuzzTest-aware test runner main for Seastar-based gtests and expose it via a new
//src/v/test_utils:fuzztesttest utility library. - Add
chunked_*model-oracle and invariant PBTs using FuzzTest (FUZZ_TEST) and wire them into the container tests BUILD. - Add Bazel module/config plumbing for FuzzTest (module dep + Bazel configs / generated rc file).
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/v/test_utils/fuzztest_main.cc | New Seastar + GTest + FuzzTest main that initializes FuzzTest and filters flags before starting Seastar’s test runner. |
| src/v/test_utils/BUILD | Adds a fuzztest test utils library that provides the new main + shared test utilities. |
| src/v/container/tests/chunked_pbt.cc | Adds FuzzTest-based PBT/oracle tests for chunked_vector and chunked_hash_map. |
| src/v/container/tests/BUILD | Adds a redpanda_cc_fuzztest target to build/run the new PBTs. |
| fuzztest.bazelrc | Adds a generated FuzzTest bazelrc (not directly imported by default). |
| bazel/test.bzl | Adds a new redpanda_cc_fuzztest helper wrapping _redpanda_cc_unit_test. |
| MODULE.bazel | Adds fuzztest Bazel module dependency. |
| MODULE.bazel.lock | Lockfile updates for the new module and transitive deps. |
| .bazelrc | Adds --config=fuzztest / --config=fuzztest-cov and related instrumentation + warning suppression settings. |
Comment on lines
+333
to
+345
| def redpanda_cc_fuzztest( | ||
| name, | ||
| timeout, | ||
| srcs = [], | ||
| defines = [], | ||
| deps = [], | ||
| args = [], | ||
| env = {}, | ||
| cpu = None, | ||
| memory = None, | ||
| data = [], | ||
| tags = []): | ||
| _redpanda_cc_unit_test( |
There was a problem hiding this comment.
redpanda_cc_fuzztest is named very similarly to the existing redpanda_cc_fuzz_test helper but behaves differently (unit-test wrapper vs libFuzzer-style target). This is easy to confuse at call sites and is inconsistent with the underscore-separated naming used by the other helpers in this file. Consider renaming to something more explicit (e.g. redpanda_cc_fuzztest_gtest / redpanda_cc_fuzztest_unit_test) or otherwise making the distinction unambiguous in the API.
Comment on lines
+379
to
+380
| build --per_file_copt=.*fuzztest.*@-Wno-nullability-completeness,-Wno-unused-parameter,-Wno-deprecated-declarations,-Wno-sign-compare | ||
| build --per_file_copt=.*_pbt\.cc@-Wno-deprecated-declarations,-Wno-sign-compare,-Wno-unused-parameter,-Wno-nullability-completeness |
There was a problem hiding this comment.
The build --per_file_copt warning suppressions are applied globally (no :fuzztest config qualifier), and the .*_pbt\.cc regex is especially broad. This risks masking real warnings in non-fuzz builds and in any future file that happens to match the pattern. Prefer scoping these suppressions to a dedicated config (e.g. build:fuzztest) and/or narrowing the regex to the specific fuzz/PBT sources, or moving the suppressions into the redpanda_cc_fuzztest macro/targets via copts so they only affect fuzztest binaries.
Suggested change
| build --per_file_copt=.*fuzztest.*@-Wno-nullability-completeness,-Wno-unused-parameter,-Wno-deprecated-declarations,-Wno-sign-compare | |
| build --per_file_copt=.*_pbt\.cc@-Wno-deprecated-declarations,-Wno-sign-compare,-Wno-unused-parameter,-Wno-nullability-completeness | |
| build:fuzztest --per_file_copt=.*fuzztest.*@-Wno-nullability-completeness,-Wno-unused-parameter,-Wno-deprecated-declarations,-Wno-sign-compare | |
| build:fuzztest --per_file_copt=.*_pbt\.cc@-Wno-deprecated-declarations,-Wno-sign-compare,-Wno-unused-parameter,-Wno-nullability-completeness |
Comment on lines
+34
to
+37
| srcs = [ | ||
| "fuzztest_main.cc", | ||
| "gtest_utils.cc", | ||
| ], |
There was a problem hiding this comment.
gtest_utils.cc is compiled into both :gtest and the new :fuzztest test utils libraries. Any test target that (directly or transitively) depends on both will hit duplicate symbol/ODR link errors for rp_test_listener/get_test_directory. To avoid this footgun, consider factoring gtest_utils.cc into a small shared cc_library and having both mains depend on that, or otherwise ensuring the two libraries cannot be combined.
dotnwat
reviewed
Apr 20, 2026
Member
dotnwat
left a comment
dotnwat
left a comment
There was a problem hiding this comment.
awesome. tagging @travisdowns here at CKO I recall a discussion about some upcoming fuzzing fixes in upstream Seastar?
andrwng
reviewed
Apr 23, 2026
Comment on lines
+101
to
+111
| for (const auto& op : ops) { | ||
| switch (op.kind % kNumVecOps) { | ||
| case 0: // push_back | ||
| impl.push_back(op.value); | ||
| oracle.push_back(op.value); | ||
| break; | ||
| case 1: // pop_back | ||
| if (!oracle.empty()) { | ||
| impl.pop_back(); | ||
| oracle.pop_back(); | ||
| } |
Contributor
There was a problem hiding this comment.
What restrictions, if any, are there around writing oracle code with ss::futures/get() calls in them? The previous commit makes me think there are no restrictions?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds Google FuzzTest infrastructure and implements some PBTs and oracle PBTs that can also be run as coverage-guided fuzz tests. See the individual commits for vibe details.
This was entirely vibed by Claude. I had it vibe the final coverage report stuff so I could verify that the fuzzing is exploring relevant codepaths. It is. It also makes it easy to figure out new test cases to hit more of the code.
Backports Required
Release Notes