cmd/runtimetest/main: Use TAP diagnostics for errors

This lets us print SHOULD violations even if they're non-fatal.  It
also gets us back to consumable TAP output.  I added a /foo entry to
defaultFS for testing, and before this commit:

  $ ./runtimetest
  TAP version 13
  ok 1 - root filesystem
  ok 2 - hostname
  ok 3 - mounts
  not ok 4 - capabilities
  ok 5 - default symlinks
  ok 6 - default devices
  ok 7 - linux devices
  not ok 8 - linux process
  ok 9 - masked paths
  ok 10 - oom score adj
  ok 11 - read only paths
  not ok 12 - rlimits
  ok 13 - sysctls
  ok 14 - uid mappings
  ok 15 - gid mappings
  1..15
  3 errors occurred:

  * Unexpected bounding capability chown set for process
  * UID expected: 1, actual: 1000
  * Wrong rlimit value:
  $ echo $?
  1

The TAP 13 spec doesn't cover script exit-code handling [1], but that
exit code confuses prove unless you set --ignore-exit:

  $ prove ./runtimetest
  ./runtimetest .. 1/? 3 errors occurred:

  * Unexpected bounding capability chown set for process
  * UID expected: 1, actual: 1000
  * Wrong rlimit value:
  ./runtimetest .. Dubious, test returned 1 (wstat 256, 0x100)
  Failed 4/16 subtests

  Test Summary Report
  -------------------
  ./runtimetest (Wstat: 256 Tests: 16 Failed: 4)
    Failed tests:  4, 6, 9, 13
    Non-zero exit status: 1
  Files=1, Tests=16,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

Note the "Dubious, test returned 1".  And with the diagnostics written
to stderr, there's no way to have prove catch that.

With this commit, we're back to TAP-compatible output and exit codes,
using TAP's diagnostics [2] to share the failure details.

  $ ./runtimetest
  TAP version 13
  ok 1 - root filesystem
  ok 2 - hostname
  ok 3 - mounts
  not ok 4 - capabilities
  # Unexpected bounding capability chown set for process
  ok 5 - default symlinks
  ok 6 - default file system
  # /foo SHOULD exist and expected type is tmpfs
  # Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config-linux.md#default-filesystems
  ok 7 - default devices
  ok 8 - linux devices
  not ok 9 - linux process
  # UID expected: 1, actual: 1000
  ok 10 - masked paths
  ok 11 - oom score adj
  ok 12 - read only paths
  not ok 13 - rlimits
  # Wrong rlimit value:
  ok 14 - sysctls
  ok 15 - uid mappings
  ok 16 - gid mappings
  1..16
  $ echo $?
  0

You can use prove without --ignore-edit to summarize:

  $ prove ./runtimetest
  ./runtimetest .. Failed 3/16 subtests

  Test Summary Report
  -------------------
  ./runtimetest (Wstat: 0 Tests: 16 Failed: 3)
    Failed tests:  4, 9, 13
  Files=1, Tests=16,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
  Result: FAIL

And prove knows that the tests failed from TAP, with the script exit
code saying "we were able to run the tests" and not speaking to
pass/fail:

  $ echo $?
  1

If that's too compact, you can ask prove to show failures and
comments:

  $ prove -fo ./runtimetest
  ./runtimetest .. 1/?
  not ok 4 - capabilities
  # Unexpected bounding capability chown set for process
  # /foo SHOULD exist and expected type is tmpfs
  # Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config-linux.md#default-filesystems
  not ok 9 - linux process
  # UID expected: 1, actual: 1000
  not ok 13 - rlimits
  # Wrong rlimit value:
  ./runtimetest .. Failed 3/16 subtests

  Test Summary Report
  -------------------
  ./runtimetest (Wstat: 0 Tests: 16 Failed: 3)
    Failed tests:  4, 9, 13
  Files=1, Tests=16,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

You can also turn that SHOULD error into a failure by cranking up the
compliance level:

  $ prove -fo ./runtimetest :: --compliance-level SHOULD
  ./runtimetest .. 1/?
  not ok 4 - capabilities
  # Unexpected bounding capability chown set for process
  not ok 6 - default file system
  # /foo SHOULD exist and expected type is tmpfs
  # Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config-linux.md#default-filesystems
  not ok 9 - linux process
  # UID expected: 1, actual: 1000
  not ok 13 - rlimits
  # Wrong rlimit value:
  ./runtimetest .. Failed 4/16 subtests

  Test Summary Report
  -------------------
  ./runtimetest (Wstat: 0 Tests: 16 Failed: 4)
    Failed tests:  4, 6, 9, 13
  Files=1, Tests=16,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

[1]: https://testanything.org/tap-version-13-specification.html#todo
[2]: https://testanything.org/tap-version-13-specification.html#diagnostics

Signed-off-by: W. Trevor King <wking@tremily.us>

Author: wking

cmd/runtimetest/main.go

@@ -14,7 +14,6 @@ import (
 	"strings"
 	"syscall"
 
-	"github.com/hashicorp/go-multierror"
 	"github.com/mndrix/tap-go"
 	rspec "github.com/opencontainers/runtime-spec/specs-go"
 	"github.com/sirupsen/logrus"
@@ -706,33 +705,27 @@ func validate(context *cli.Context) error {
 		complianceLevel = ociErr.ComplianceMust
 		logrus.Warningf("%s, using 'MUST' by default.", err.Error())
 	}
-	var validationErrors error
-	for _, v := range defaultValidations {
-		err := v.test(spec)
-		t.Ok(err == nil, v.description)
-		if err != nil {
-			if e, ok := err.(*ociErr.Error); ok && e.Level < complianceLevel {
-				continue
-			}
-			validationErrors = multierror.Append(validationErrors, err)
-		}
+	validations := defaultValidations
+	if platform == "linux" {
+		validations = append(validations, linuxValidations...)
 	}
 
-	if platform == "linux" {
-		for _, v := range linuxValidations {
-			err := v.test(spec)
-			t.Ok(err == nil, v.description)
-			if err != nil {
-				if e, ok := err.(*ociErr.Error); ok && e.Level < complianceLevel {
-					continue
-				}
-				validationErrors = multierror.Append(validationErrors, err)
+	for _, v := range validations {
+		err := v.test(spec)
+		if err == nil {
+			t.Pass(v.description)
+		} else {
+			if e, ok := err.(*ociErr.Error); ok {
+				t.Ok(e.Level < complianceLevel, v.description)
+			} else {
+				t.Fail(v.description)
 			}
+			t.Diagnostic(err.Error())
 		}
 	}
 	t.AutoPlan()
 
-	return validationErrors
+	return nil
 }
 
 func main() {