tests: validate plugin enumeration contract

[SchmeatMilk]

Summary

Reworked from the closed #1962 per maintainer feedback: this now validates the contract of enumerate_plugins() instead of re-implementing garak's discovery logic.

The contract under test: every concrete plugin class in a category (probes, detectors) is returned by enumerate_plugins(), and base classes are deliberately excluded (the .base. name skip). The test collects concrete plugin classes reachable from the category base classes via the normal Python subclass registry, filters to the garak.<category>. namespace (matching what discovery enumerates), and asserts each is enumerated — without duplicating the module-walking discovery code.

• tests/plugins/test_plugin_enumeration_coverage.py
  • test_all_concrete_plugin_classes_are_enumerated[detectors|probes]: every concrete class in the namespace is in enumerate_plugins().
  • test_base_classes_excluded_from_enumeration[detectors|probes]: confirms base classes are correctly not enumerated (guards against the contract regressing the other way).

Why this is not the previous (rejected) approach

#1962 mirrored _enumerate_plugin_klasses discovery by re-scanning modules — the reviewer correctly noted that duplicates implementation and adds maintenance cost. This version calls only the public enumerate_plugins() and uses the subclass registry to know what should be there, so it tests behaviour, not internals.

Why not a duplicate of an existing PR

Issue #713 is unassigned with no linked open PR (gh pr list --search "713 in:body" returns only unrelated #975). Confirmed no open PR addresses plugin enumeration coverage.

Test plan

• pytest tests/plugins/test_plugin_enumeration_coverage.py -> 4 passed (0.2s).
• Negative control proven: monkeypatching enumerate_plugins to drop detectors.always.Pass makes test_all_concrete_plugin_classes_are_enumerated[detectors] FAIL with:

  concrete plugin classes in garak.detectors not picked up by plugin enumeration (see issue tests: check plugin modules for classes that aren't picked up by plugin enumeration #713): ['detectors.always.Pass']

• Pre-existing failure unrelated to this change (optional audio deps missing): test_instantiate_probes[probes.audio.AudioAchillesHeel]. Not touched by this PR.

AI assistance disclosure

Developed with AI assistance (agent-authored), then verified locally by running the suite and the negative control. Approach revised in direct response to the #1962 review.

Co-authored-by: Malik Baptiste mbaptiste20@gmail.com

[jmartin-tech]

This PR again does not test the contract, it attempts to reimplement the patterns in the existing method.

Some of the tests here are almost valid, as validating that no base classes or classes that are not concrete would be part of the contract for enumerate_plugins()

A valid test set would validate that the behavior intended behavior of enumerate_plugins() is met. Those criteria being:

• ALL plugins types that enumerate_plugins returns will represent concrete classes that extend the base of that plugin type and are not abstract
• any newly added plugin class injected into the paths enumerated is properly added
• any plugin class removed from the paths enumerated is properly removed

These will not be simple tests to implement as they will need to manipulate the testing runtime environment and ensure they do not leave behind invalid state or leave the filesystem in a state that might cause a user to accidentally commit testing artifacts during the development or package build even when tests are terminated prematurely or due to some other common runtime issue.

Note all commits currently on this branch fail to meet DCO requirement. If you would like to continue to iterate on this goal be sure to clean up the commit history and ensure only properly formatted commits exist in the branch.