feat: add chaos testing module for fault injection

[ybdarrenwang]

Description

Introduces a chaos testing module for fault injection during agent evaluation. Enables systematic testing of agent resilience under tool failures and response corruption without modifying agent code.

Key capabilities:

• Effect hierarchy — chaos effects split into pre-hook (cancel tool call with error: Timeout, NetworkError, ExecutionError, ValidationError) and post-hook (corrupt tool response: TruncateFields, RemoveFields, CorruptValues), each with effect-specific parameters (e.g., corrupt_ratio, max_length, remove_ratio)
• ChaosCase — extends Case with an effects field that carries the failure injection config. Provides ChaosCase.expand(cases, effect_maps) to generate the Cartesian product of base cases × named effect maps.
• Plugin integration — ChaosPlugin hooks into Strands' native BeforeToolCallEvent/AfterToolCallEvent system; reads the active ChaosCase from a ContextVar (zero chaos concepts in user task code)
• Experiment runner — ChaosExperiment composes the base Experiment to run ChaosCase objects, managing ContextVar lifecycle per case for thread/async safety

Related Issues

#114

Documentation PR

strands-agents/docs#836

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Review Summary

Assessment: Request Changes

This PR introduces a well-structured chaos testing module with clean separation between effects, scenarios, plugin, and experiment orchestration. The ContextVar-based design for concurrency safety and the composition with the existing Experiment class are solid architectural decisions.

Review Categories

• Unimplemented Feature: apply_rate is declared as a user-facing parameter but never checked anywhere in the execution path — this will confuse users who set it expecting probabilistic firing.
• Documentation Accuracy: Multiple docstrings reference non-existent classes (Timeout, NetworkError) instead of the actual ToolCallFailure API. These should be corrected before merge to avoid misleading users.
• Style Guide Compliance: All logger.info(f"...") calls should use lazy %s formatting per the project's STYLE_GUIDE.
• Type Safety: The hasattr(effect, "max_length") duck-typing in the plugin should use an explicit isinstance check or a proper polymorphic dispatch.
• API Review: This PR introduces new public abstractions (ChaosExperiment, ChaosPlugin, ChaosScenario, effect hierarchy) that customers will use directly. Consider adding the needs-api-review label per the API Bar Raising process.

The overall design is thoughtful and the test coverage is good. Addressing the apply_rate gap and the doc inaccuracies are the most impactful changes needed.

[github-actions]

Review Summary (Follow-up)

Assessment: Comment

All feedback from the previous review has been thoroughly addressed — apply_rate is wired up, docstrings are accurate, logging follows the style guide, type-safety is improved, and the task wrapper is deduplicated. Nice work on the quick turnaround.

Remaining Items

• Async task support: _wrap_task returns a sync wrapper unconditionally, which breaks async tasks passed to run_evaluations_async. Needs an async-aware branch.
• Input validation: run_evaluations should reject async tasks with a clear error (matching base Experiment behavior) rather than silently producing bad results.

These are straightforward fixes. Once addressed, this looks good to merge.

[github-actions]

Review Summary (Round 3)

Assessment: Request Changes

All prior feedback has been resolved. One important serialization issue remains that would cause data loss in error reporting and persistence scenarios.

Details

• Serialization: ChaosCase.effects uses the abstract base type ChaosEffect in its annotation, causing Pydantic to drop concrete fields during model_dump() and fail on model_validate(). Needs a discriminated union.
• Test coverage: The async task path in _wrap_task lacks a corresponding test.

The architecture, ContextVar design, plugin implementation, and test structure are all solid.

[github-actions]

Review Summary (Round 4)

Assessment: Approve

All previously identified issues have been thoroughly addressed: discriminated union serialization is working correctly, async task handling is properly implemented with tests, apply_rate was cleanly removed, and the code is well-tested (81 tests passing).

Minor items noted

• Docstring accuracy: One stale reference to "ToolCallFailure" in plugin.py
• Style consistency: Optional import can be replaced with | None syntax (project requires Python 3.10+)

The architecture is solid — ContextVar lifecycle, discriminated union serialization, plugin hooks, and the composition pattern with base Experiment are all well-designed.