-
Notifications
You must be signed in to change notification settings - Fork 794
REFACTOR: Migrate CLI to use pyrit.models #1997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rlundeen2
wants to merge
5
commits into
microsoft:main
Choose a base branch
from
rlundeen2:rlundeen2/phase-16-pyrit-models-plan
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,266
−767
Open
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
e72af5f
REFACTOR: CLI and backend consume typed pyrit.models catalog types (P…
rlundeen2 88091b0
Merge remote-tracking branch 'origin/main' into rlundeen2/phase-16-py…
rlundeen2 8bdbef7
Update pyrit/models/catalog/__init__.py
rlundeen2 a7ff7d2
Merge remote-tracking branch 'origin/main' into rlundeen2/phase-16-py…
rlundeen2 2746c7d
Restore per-field descriptions on scenario/initializer catalog models
rlundeen2 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,48 +2,23 @@ | |
| # Licensed under the MIT license. | ||
|
|
||
| """ | ||
| Scenario API response models. | ||
| REST envelopes for the scenario endpoints. | ||
|
|
||
| Scenarios are multi-attack security testing campaigns. These models represent | ||
| the metadata about available scenarios (listing) and scenario execution (runs). | ||
| Canonical scenario catalog/run types (``RegisteredScenario``, | ||
| ``ScenarioParameterSummary``, ``ScenarioRunSummary``, ``RunScenarioRequest``) | ||
| live in ``pyrit.models.catalog.scenario`` and should be imported from there | ||
| directly. | ||
| """ | ||
|
|
||
| from datetime import datetime | ||
| from enum import Enum | ||
| from typing import Any | ||
|
|
||
| from pydantic import BaseModel, Field | ||
|
|
||
| from pyrit.backend.models.common import PaginationInfo | ||
| from pyrit.models.catalog.scenario import RegisteredScenario, ScenarioRunSummary | ||
|
|
||
|
|
||
| class ScenarioParameterSummary(BaseModel): | ||
| """Summary of a scenario-declared parameter.""" | ||
|
|
||
| name: str = Field(..., description="Parameter name (e.g., 'max_turns')") | ||
| description: str = Field(..., description="Human-readable description of the parameter") | ||
| default: str | None = Field(None, description="Default value as a display string, or None if required") | ||
| param_type: str = Field(..., description="Type of the parameter as a display string (e.g., 'int', 'str')") | ||
| choices: list[str] | None = Field(None, description="Allowed values as strings, or None if unconstrained") | ||
| is_list: bool = Field(False, description="True when the parameter accepts a list of values (e.g., list[str])") | ||
|
|
||
|
|
||
| class RegisteredScenario(BaseModel): | ||
| """Summary of a registered scenario.""" | ||
|
|
||
| scenario_name: str = Field(..., description="Scenario name (e.g., 'foundry.red_team_agent')") | ||
| scenario_type: str = Field(..., description="Scenario type identifier (e.g., 'RedTeamAgentScenario')") | ||
| description: str = Field(..., description="Human-readable description of the scenario") | ||
| default_strategy: str = Field(..., description="Default strategy name used when none specified") | ||
| aggregate_strategies: list[str] = Field( | ||
| ..., description="Aggregate strategies that combine multiple attack approaches" | ||
| ) | ||
| all_strategies: list[str] = Field(..., description="All available concrete strategy names") | ||
| default_datasets: list[str] = Field(..., description="Default dataset names used by the scenario") | ||
| max_dataset_size: int | None = Field(None, description="Maximum items per dataset (None means unlimited)") | ||
| supported_parameters: list[ScenarioParameterSummary] = Field( | ||
| default_factory=list, description="Scenario-declared custom parameters" | ||
| ) | ||
| __all__ = [ | ||
| "ListRegisteredScenariosResponse", | ||
| "ScenarioRunListResponse", | ||
| ] | ||
|
|
||
|
|
||
| class ListRegisteredScenariosResponse(BaseModel): | ||
|
|
@@ -53,73 +28,6 @@ class ListRegisteredScenariosResponse(BaseModel): | |
| pagination: PaginationInfo = Field(..., description="Pagination metadata") | ||
|
|
||
|
|
||
| # ============================================================================ | ||
| # Scenario Run Models | ||
| # ============================================================================ | ||
|
|
||
|
|
||
| class ScenarioRunStatus(str, Enum): | ||
| """Status of a scenario run, aligned with core ScenarioRunState.""" | ||
|
|
||
| CREATED = "CREATED" | ||
| INITIALIZING = "INITIALIZING" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. noting that GHC noticed |
||
| IN_PROGRESS = "IN_PROGRESS" | ||
| COMPLETED = "COMPLETED" | ||
| FAILED = "FAILED" | ||
| CANCELLED = "CANCELLED" | ||
|
|
||
|
|
||
| class RunScenarioRequest(BaseModel): | ||
| """Request body for starting a scenario run.""" | ||
|
|
||
| scenario_name: str = Field(..., description="Scenario name (e.g., 'foundry.red_team_agent')") | ||
| target_name: str = Field(..., description="Name of a registered target from the TargetRegistry") | ||
| initializers: list[str] | None = Field( | ||
| None, description="Initializer names to run before scenario (e.g., ['target', 'load_default_datasets'])" | ||
| ) | ||
| strategies: list[str] | None = Field(None, description="Strategy names to use (uses scenario default if omitted)") | ||
| dataset_names: list[str] | None = Field(None, description="Dataset names to use (uses scenario default if omitted)") | ||
| max_dataset_size: int | None = Field(None, ge=1, description="Maximum items per dataset") | ||
| max_concurrency: int = Field(10, ge=1, le=100, description="Maximum concurrent operations") | ||
| max_retries: int = Field(0, ge=0, le=20, description="Maximum retry attempts on failure") | ||
| labels: dict[str, str] | None = Field(None, description="Labels to attach to memory entries") | ||
| scenario_params: dict[str, Any] | None = Field( | ||
| None, | ||
| description="Custom parameters for the scenario (passed to scenario.set_params_from_args). " | ||
| "Keys are parameter names declared by the scenario's supported_parameters().", | ||
| ) | ||
| initializer_args: dict[str, dict[str, Any]] | None = Field( | ||
| None, | ||
| description="Per-initializer arguments keyed by initializer name. " | ||
| "Each value is a dict of args passed to that initializer's set_params_from_args(). " | ||
| "Example: {'target': {'endpoint': 'https://...'}}.", | ||
| ) | ||
| scenario_result_id: str | None = Field( | ||
| None, | ||
| description="Optional ID of an existing ScenarioResult to resume. " | ||
| "If provided, the scenario will resume from prior progress instead of starting fresh.", | ||
| ) | ||
|
|
||
|
|
||
| class ScenarioRunSummary(BaseModel): | ||
| """Response for a scenario run (status + result details).""" | ||
|
|
||
| scenario_result_id: str = Field(..., description="UUID of the ScenarioResult in memory") | ||
| scenario_name: str = Field(..., description="Registry key of the scenario being run") | ||
| scenario_version: int = Field(0, ge=0, description="Version of the scenario") | ||
| status: ScenarioRunStatus = Field(..., description="Current run status") | ||
| created_at: datetime = Field(..., description="When the run was created") | ||
| updated_at: datetime = Field(..., description="When the run status last changed") | ||
| error: str | None = Field(None, description="Error message if status is FAILED") | ||
| error_type: str | None = Field(None, description="Exception class name if status is FAILED") | ||
| strategies_used: list[str] = Field(default_factory=list, description="Strategy names that were executed") | ||
| total_attacks: int = Field(0, ge=0, description="Total number of attack results persisted for this run") | ||
| completed_attacks: int = Field(0, ge=0, description="Number of attacks that reached a terminal outcome") | ||
| objective_achieved_rate: int = Field(0, ge=0, le=100, description="Success rate as percentage (0-100)") | ||
| labels: dict[str, str] = Field(default_factory=dict, description="Labels attached to this run") | ||
| completed_at: datetime | None = Field(None, description="When the scenario finished") | ||
|
|
||
|
|
||
| class ScenarioRunListResponse(BaseModel): | ||
| """Response for listing scenario runs.""" | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
flagging a small divergence from
pyrit/backend/models/{scenarios,initializers,targets}.py keep these as re-export shims for one release.mentioned in the gist (i.e., something likefrom pyrit.backend.models import RegisteredScenariowill break for without DeprecationWarning.)