Enabling the use of a Real test runner like CTest would bring in so many advantages:
- Let one hone in on failed tests more quickly: one current failure mode is for the very last test in the suite to fail, requiring running everything before it before we can get to the test we actually care about.
- Running the test suite in parallel would be hugely helpful, especially for catching failures more quickly.
- CTest has smarter and better output, in that it's far less verbose (and thus doesn't drown the first test failing in “ok well the other 100 tests after it passed, have fun scrolling now~”)
- The testing options (“run external tests”, etc.) don't need to be passed at configure time (which is a little weird), but instead passed to CTest.
Of course, we could “just” improve our own test runners, but c'mon. Anyone else, feel free to go do that—but I already dislike the current runners' complexity enough, lol. I'd rather not reinvent those wheels.
I'm rejecting the specific implementation in #1539 on the grounds that it duplicates our test logic between the existing “bulk“ runners and the new “piecemeal” runners. But it does have some good ideas, such as creating some library of functions that all runner scripts should source. Why haven't we thought of that earlier!?
As for the architecture of integrating with CTest, well, a major requirement is that we want it to be optional, and thus the Bash scripts have to be self-sufficient. By the rule of a single source of truth, it follows that the scripts have to contain the list of all tests, and their specific logic. Thankfully, CMake is versatile enough that we can invoke arbitrary code at configure time; thus, if we add some introspection logic to the Bash scripts so they can emit the list of test cases, and we give them the ability to only run a single test at a time, then we can have the following:
- CMake invokes
test/runner.sh --list-tests
- gets back some list (CSV or whatever, we could even emit CMake code directly?)
- generates CTest cases based on those
- ...which each invoke e.g.
test/asm/runner.sh <test name>
The action list thus looks like this:
Enabling the use of a Real test runner like CTest would bring in so many advantages:
Of course, we could “just” improve our own test runners, but c'mon. Anyone else, feel free to go do that—but I already dislike the current runners' complexity enough, lol. I'd rather not reinvent those wheels.
I'm rejecting the specific implementation in #1539 on the grounds that it duplicates our test logic between the existing “bulk“ runners and the new “piecemeal” runners. But it does have some good ideas, such as creating some library of functions that all runner scripts should source. Why haven't we thought of that earlier!?
As for the architecture of integrating with CTest, well, a major requirement is that we want it to be optional, and thus the Bash scripts have to be self-sufficient. By the rule of a single source of truth, it follows that the scripts have to contain the list of all tests, and their specific logic. Thankfully, CMake is versatile enough that we can invoke arbitrary code at configure time; thus, if we add some introspection logic to the Bash scripts so they can emit the list of test cases, and we give them the ability to only run a single test at a time, then we can have the following:
test/runner.sh --list-teststest/asm/runner.sh <test name>The action list thus looks like this:
test/utils.sh