Revamp code coverage tooling by dotnwat · Pull Request #30181 · redpanda-data/redpanda

dotnwat · 2026-04-15T20:58:29Z

Revamp code coverage tooling

HTML, terminal, and LLM optimized output formats
Testing for "what's coverage like for my PR/diff"
Testing of the coverage tooling itself to avoid bit rot
Claude skill for improving coverage
Some other junk

Coverage: //src/v/cloud_topics/level_one/...

  Totals:  Lines: 66816/292585 (22.8%)  Functions: 21443/103109 (20.8%)  Branches: 11583/72346 (16.0%)

  Scope: src/v/cloud_topics/level_one/

  File                                                             Lines      Functions       Branches
  ─────────────────────────────────────────────────────────────────────────────────────────────────────
  ...v/cloud_topics/level_one/domain/db_domain_manager.cc  1041/1686  61.7%    45/51  88.2%  297/496  59.9%
  ...d_topics/level_one/metastore/replicated_metastore.cc  535/892  60.0%    31/41  75.6%  136/234  58.1%
  ...oud_topics/level_one/domain/simple_domain_manager.cc  393/688  57.1%    21/33  63.6%   93/174  53.4%
  ...cloud_topics/level_one/metastore/lsm/state_update.cc  932/1126  82.8%    18/28  64.3%  314/380  82.6%
  ...cloud_topics/level_one/metastore/lsm/state_reader.cc  445/591  75.3%    27/30  90.0%  141/176  80.1%
  ...ud_topics/level_one/metastore/partition_validator.cc  323/451  71.6%     9/12  75.0%  112/188  59.6%
  src/v/cloud_topics/level_one/common/file_io.cc            97/224  43.3%    16/22  72.7%    16/56  28.6%
  src/v/cloud_topics/level_one/metastore/leader_router.cc  393/515  76.3%   84/125  67.2%    29/56  51.8%
  src/v/cloud_topics/level_one/common/object.cc            472/582  81.1%    87/94  92.6%   85/118  72.0%
  ...topics/level_one/frontend_reader/level_one_reader.cc  301/405  74.3%    17/18  94.4%   83/112  74.1%
  src/v/cloud_topics/level_one/compaction/sink.cc          226/314  72.0%    16/16 100.0%    45/72  62.5%
  ...ud_topics/level_one/metastore/leader_router_probe.cc     5/85   5.9%     1/18   5.6%      1/2  50.0%
  ..._topics/level_one/metastore/lsm/garbage_collector.cc  110/190  57.9%      4/6  66.7%    45/86  52.3%
  ...cloud_topics/level_one/metastore/simple_metastore.cc  615/688  89.4%    45/45 100.0%  181/222  81.5%
  src/v/cloud_topics/level_one/metastore/rpc_types.h         16/85  18.8%     9/39  23.1%    10/16  62.5%
  ...ud_topics/level_one/compaction/log_info_collector.cc  162/228  71.1%    10/12  83.3%    40/70  57.1%
  ...loud_topics/level_one/metastore/lsm/replicated_db.cc  280/337  83.1%    17/17 100.0%    76/98  77.6%
  src/v/cloud_topics/level_one/metastore/state_update.cc   716/772  92.7%    21/22  95.5%  234/262  89.3%
  src/v/cloud_topics/level_one/metastore/service.cc          18/70  25.7%     5/18  27.8%      0/0     -
  src/v/cloud_topics/level_one/metastore/topic_purger.cc   122/169  72.2%    16/19  84.2%    29/48  60.4%
  ─────────────────────────────────────────────────────────────────────────────────────────────────────
  Showing 20 of 97 files (sorted by uncovered)
  4 files with 0% coverage (use -f <pattern> to inspect)
  1872 out-of-scope files hidden (use --all-files to show)

Backports Required

Release Notes

none

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Copilot

Pull request overview

This PR replaces legacy coverage utilities with a unified tools/run-cov tool that can generate terminal/HTML/LLM-friendly coverage reports, including coverage focused on changed lines in a diff, and adds Bazel-backed tests and fixtures to prevent regressions.

Changes:

Add tools/run-cov (Python) to run/reuse Bazel C++ coverage and emit terminal, HTML, and LLM-optimized reports (including diff coverage).
Add Bazel sh_test coverage-tool regression tests plus LCOV/diff fixtures under tools/tests/.
Remove legacy coverage scripts and update repo config (.bazelrc coverage settings, .gitignore for coverage-out/).

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`tools/run-cov`	New unified coverage runner/report generator (terminal/html/llm + diff coverage).
`tools/tests/run_cov_test.sh`	End-to-end Bazel shell test for `tools/run-cov` outputs and failure modes.
`tools/tests/BUILD`	Registers the `run_cov_test` Bazel test and its runfiles.
`tools/tests/testdata/sample.diff`	Fixture unified diff used to validate diff-coverage classification.
`tools/tests/testdata/coverage_fixture.dat`	Fixture LCOV data used by the regression test.
`tools/tests/testdata/BUILD`	Exposes test fixtures as Bazel data files.
`tools/BUILD`	Exports `run-cov` for Bazel runfiles usage.
`.bazelrc`	Updates coverage config flags/environment.
`.gitignore`	Ignores `coverage-out/` (default output dir).
`.claude/skills/improve-coverage/SKILL.md`	Adds a Claude skill/playbook for using the new coverage tooling.
`tools/single_test_cov.sh`	Removed legacy single-test coverage script.
`tools/gen_coverage.py`	Removed legacy coverage generator.
`tools/coverage_dash.py`	Removed legacy coverage dashboard generator.

Copilot · 2026-04-20T18:35:01Z

+                info.changed_lines[current_file].add(current_line)
+                current_line += 1
+            elif line.startswith("-"):
+                pass  # deleted line — don't advance new-file counter


parse_unified_diff() treats any non "+"/"-" line inside a hunk as a context line and increments current_line. This will miscount when the diff contains metadata lines like \ No newline at end of file (which should not advance either side’s line counters), causing off-by-one changed-line mapping for the remainder of the hunk. Consider explicitly skipping lines starting with \\ (and potentially other non-hunk metadata) when current_file is set.

Suggested change

pass # deleted line — don't advance new-file counter

pass # deleted line — don't advance new-file counter

elif line.startswith("\\"):

pass # hunk metadata (for example: '\ No newline at end of file')

vbotbuildovich · 2026-04-20T19:48:21Z

CI test results

test results on build#83411

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	IcebergUsageTest	test_iceberg_usage	{"catalog_type": "rest_hadoop", "cloud_storage_type": 1, "query_engine": "spark"}	integration	https://buildkite.com/redpanda/redpanda/builds/83411#019dac33-7f0d-477e-8ac8-a75966eee234	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0038, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=IcebergUsageTest&test_method=test_iceberg_usage

StephanDollberg · 2026-04-21T08:37:43Z

+        return None
+
+    bin_dir = os.path.join(
+        output_base, "external", "current_llvm_toolchain_llvm", "bin"


Did you try this on arm. I remember something breaking there when I did similar for PGO.

I didn't. I don't think any developers work on arm. I guess if we wanted to use this tooling in CI we'd need to address that for sure.

StephanDollberg · 2026-04-21T08:40:12Z

+    workspace_root: str | None,
+    output,
+):
+    """Generate an LLM-optimized diff coverage report in markdown."""


What does "LLM-optimized" mean?

basically it means no formatting.

StephanDollberg · 2026-04-21T08:51:06Z

+# ---------------------------------------------------------------------------
+
+
+def parse_lcov(path: str) -> CoverageReport:


Bit confused by all the things going on in this script:

Parsing lcov

Demangling

Output generation

Isn't there existing tooling for all of this? Gcov? Never really been much into coverage reports.

I'm unaware of any tooling that would provide reporting on the terminal. But maybe? If there is then we should use that.

dotnwat · 2026-04-20T17:10:32Z

-build:coverage --action_env=BAZEL_USE_LLVM_NATIVE_COVERAGE=1
-build:coverage --action_env=GCOV=llvm-profdata
-build:coverage --copt=-DNDEBUG
-build:coverage --define=dynamic_link_tests=true
+build:coverage --repo_env=BAZEL_USE_LLVM_NATIVE_COVERAGE=1
 build:coverage --combined_report=lcov
 build:coverage --experimental_use_llvm_covmap
 build:coverage --experimental_generate_llvm_lcov
-build:coverage --experimental_split_coverage_postprocessing
-build:coverage --experimental_fetch_all_coverage_outputs
-build:coverage --collect_code_coverage


i copied these settings from another c++ project two years ago. i don't know how much of that is old bazel versions vs secret magic settings. in any case, with these simplified settings coverage passes the spot testing i've been doing.

dotnwat · 2026-04-20T17:53:09Z

+
+
+@dataclass
+class LineCoverage:


The tool is large because it does parsing of coverage data and commits in order to be able to build custom output formats for UIs like the terminal.

dotnwat · 2026-04-21T16:50:41Z

+    workspace_root: str | None,
+    output,
+):
+    """Generate an LLM-optimized diff coverage report in markdown."""


basically it means no formatting.

dotnwat · 2026-04-21T16:51:43Z

+# ---------------------------------------------------------------------------
+
+
+def parse_lcov(path: str) -> CoverageReport:


I'm unaware of any tooling that would provide reporting on the terminal. But maybe? If there is then we should use that.

dotnwat · 2026-04-21T16:52:30Z

+        return None
+
+    bin_dir = os.path.join(
+        output_base, "external", "current_llvm_toolchain_llvm", "bin"


I didn't. I don't think any developers work on arm. I guess if we wanted to use this tooling in CI we'd need to address that for sure.

pgellert

pretty cool

pgellert · 2026-04-29T17:11:00Z

@@ -1,289 +0,0 @@
-import argparse


We should update the bazel wiki's coverage subheader to point to this new tool once this merges

github-actions Bot added the area/build label Apr 15, 2026

tools: remove previous generation of coverage tools

2639b8f

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

dotnwat force-pushed the coverage branch from 06de605 to 3658eaa Compare April 17, 2026 20:11

dotnwat added 3 commits April 20, 2026 10:47

tools: add run-cov coverage testing tool

87809d7

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

tools: add test for run-cov tool

8ccd42a

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

claude: add skill for code coverage

0bd9c1f

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

dotnwat force-pushed the coverage branch from 3658eaa to 0bd9c1f Compare April 20, 2026 17:50

dotnwat changed the title ~~[WIP] Revamp code coverage tooling~~ Revamp code coverage tooling Apr 20, 2026

dotnwat marked this pull request as ready for review April 20, 2026 18:28

Copilot AI review requested due to automatic review settings April 20, 2026 18:28

dotnwat requested a review from a team as a code owner April 20, 2026 18:28

dotnwat requested review from PrzemekZglinicki and removed request for a team April 20, 2026 18:28

Copilot started reviewing on behalf of dotnwat April 20, 2026 18:28 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

dotnwat requested review from StephanDollberg, WillemKauf, andrwng, bashtanov and pgellert and removed request for PrzemekZglinicki April 20, 2026 18:54

StephanDollberg reviewed Apr 21, 2026

View reviewed changes

dotnwat commented Apr 29, 2026

View reviewed changes

dotnwat requested a review from StephanDollberg April 29, 2026 14:27

StephanDollberg approved these changes Apr 29, 2026

View reviewed changes

pgellert approved these changes Apr 29, 2026

View reviewed changes

dotnwat merged commit ef5fa67 into redpanda-data:dev Apr 29, 2026
25 checks passed

		# ---------------------------------------------------------------------------


		def parse_lcov(path: str) -> CoverageReport:



		@dataclass
		class LineCoverage:

Conversation

dotnwat commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

vbotbuildovich commented Apr 20, 2026

CI test results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgellert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dotnwat commented Apr 15, 2026 •

edited

Loading