restore coveralls.io test coverage reporting#22700
restore coveralls.io test coverage reporting#22700cburroughs merged 13 commits intopantsbuild:mainfrom
Conversation
cburroughs
added
the
release-notes:not-required
cburroughs
force-pushed
the
csb/coveralls
branch
from
565484e to
31da088
Compare
cburroughs
force-pushed
the
csb/coveralls
branch
from
31da088 to
d70ebbb
Compare
cburroughs
changed the title
|
Example PR report: https://coveralls.io/builds/75741961 
|
|
Neat! I find those 20-something numbers surprising. Every integration test should cover a lot of the core and the relevant backend, at least. I wonder if we should run unit tests separately and only measure coverage on them? That would be for a followup of course. |
| - test_python_macos13_x86_64 | ||
| - classify_changes | ||
| runs-on: | ||
| - ubuntu-22.04 |
There was a problem hiding this comment.
Nope, it is what our generator script spit out.
| name: logs-python-test-Linux-ARM64 | ||
| overwrite: 'true' | ||
| path: .pants.d/workdir/*.log | ||
| - continue-on-error: true |
There was a problem hiding this comment.
Confirming: This means coveralls can't break a build, right?
There was a problem hiding this comment.
Not quite, that one is to satisfy the GitHub Action rules so that we upload coverage even if one of the test fails so we can see what code was executed when the test failed.
I added fail-on-error: false which should accomplish the "can't break a build" run.
sureshjoshi
left a comment
sureshjoshi
left a comment
There was a problem hiding this comment.
Approving assuming this doesn't have any significant impact on CI time (which I can't imagine it does compared to how long CI takes) - and that it failing for any reason won't break our CI... We have enough of that going around recently.
The coverage.io reporting was restored in pantsbuild#22700. It was run mostly successfully for a few weeks now, so let's add pretty coverage badge.
The coverage.io reporting was restored in #22700. It was run mostly successfully for a few weeks now, so let's add pretty coverage badge. <img width="499" height="133" alt="image" src="https://github.com/user-attachments/assets/5a2d3e81-b9db-4fd2-a3c1-50912d68ecd0" />
3 participants
coveralls.io is a (free for open source) service for visualization and reporting code coverage. It cane do basic things like chart coverage over time, aggregate statistics, and visualize individual files. The Pants project appears to have had at least some nascent integration circa 2021 and when I logged in the
pantsbuild/ pantsrepo was already here. This PR re-enabled integration using their stock GitHub Action: https://github.com/coverallsapp/github-action90% of the complication & code has to due with how we do test sharding. Because we shard the tests we need to both report the coverage for each shard, name the test suites, and then setup the GitHub
needsdependencies so that we can tell coveralls when we are "done". That required some rejiggering so that jobs know their own id.Next steps after this lands and all is well: