feat(sandbox): add Amazon Bedrock AgentCore Runtime backend

docs/running-benchmarks.md

@@ -347,6 +347,7 @@ The **Harvey LAB harness** agent is special — it runs Harvey LAB's own agent l
 | Apple Container | `--sandbox apple-container` | Local Apple Silicon macOS runs without Docker Desktop |
 | Daytona | `--sandbox daytona` | Cloud runs with concurrency (needs `DAYTONA_API_KEY`) |
 | Modal | `--sandbox modal` | Serverless, high concurrency (needs Modal auth) |
+| AgentCore | `--sandbox agentcore` | AWS-native isolated microVMs (needs AWS credentials) |
 
 Apple Container requires Apple Container 1.1+ on Apple Silicon and runs the model
 proxy inside each VM. It supports public-network, single-container arm64 tasks and
@@ -356,6 +357,104 @@ running concurrent BenchFlow processes, because the macOS allocation leak is
 system-wide. Use Docker, Daytona, or Modal for `network_mode = "no-network"`,
 multi-service, snapshot, or high-concurrency runs.
 
+### Amazon Bedrock AgentCore
+
+`--sandbox agentcore` runs each rollout in an AgentCore Runtime microVM.
+Install the extra and point BenchFlow at an execution role:
+
+```bash
+pip install 'benchflow[sandbox-agentcore]'
+export BENCHFLOW_AGENTCORE_ROLE_ARN="arn:aws:iam::<account>:role/<role>"
+export BENCHFLOW_AGENTCORE_REGION="us-west-2"   # optional, this is the default
+```
+
+The role must be assumable by `bedrock-agentcore.amazonaws.com` and able to
+pull from ECR and write CloudWatch logs. Your own credentials additionally need
+`bedrock-agentcore:*` and ECR push permissions.
+
+Unlike the other backends, AgentCore cannot take a container image directly:
+BenchFlow builds the task image for `linux/arm64`, pushes it to ECR (repository
+`benchflow-agentcore`, override with `BENCHFLOW_AGENTCORE_ECR_REPOSITORY`), and
+registers it as an agent runtime. The image is tagged by content digest, so a
+task whose environment has not changed reuses the pushed layers and the
+existing runtime instead of paying for a rebuild.
+
+#### Building without Docker
+
+By default BenchFlow builds with a local Docker daemon when one is running,
+and otherwise builds on **AWS CodeBuild** — so the backend works on a machine
+with no container runtime at all, and from CI or Windows. Set a CodeBuild
+service role to enable the remote path:
+
+```bash
+export BENCHFLOW_AGENTCORE_CODEBUILD_ROLE_ARN="arn:aws:iam::<account>:role/<build-role>"
+```
+
+That role must be assumable by `codebuild.amazonaws.com` and able to push to
+ECR, read the build bucket, and write CloudWatch logs. Your own credentials
+need `codebuild:*` and S3 access. BenchFlow creates the CodeBuild project and
+an `benchflow-agentcore-build-<account>-<region>` bucket on first use (build
+contexts expire after a day); override the bucket with
+`BENCHFLOW_AGENTCORE_BUILD_BUCKET`.
+
+Builds run on a Graviton worker, so `linux/arm64` is native — no qemu. Force
+either strategy with `BENCHFLOW_AGENTCORE_BUILDER=docker|codebuild|auto`.
+A remote build takes roughly 40s versus roughly 20s locally on Apple silicon;
+either way it happens once per distinct task image, not once per rollout.
+
+BenchFlow also appends a small HTTP responder to the image and makes it the
+entrypoint. AgentCore refuses command execution for a session whose container
+does not answer `GET /ping` on port 8080, and task images know nothing about
+AgentCore, so this shim is what makes an ordinary task image runnable.
+
+Constraints: `linux/arm64` only, single container (no compose/multi-service
+tasks), no snapshot support, and `network_mode = "no-network"` is **not**
+enforceable — AgentCore's network mode is either `PUBLIC` or `VPC`, so
+BenchFlow refuses no-network tasks on this backend rather than running them
+unisolated. The model proxy runs inside the sandbox, as on Daytona and Modal.
+Sessions default to a 15-minute idle timeout and an 8-hour lifetime; override
+with `BENCHFLOW_AGENTCORE_IDLE_TIMEOUT_SEC` / `BENCHFLOW_AGENTCORE_MAX_LIFETIME_SEC`
+if agent turns are long enough to risk reclamation mid-run.
+
+**Running a large matrix.** A rollout is a *session*, not a runtime: one
+registered runtime hosts many concurrent, filesystem-isolated microVMs. The
+account quotas point the same way — 5,000 concurrent *Active Session Workloads*
+against only 100 *Total Agents*, with `CreateAgentRuntime` limited to 5/s. So
+BenchFlow builds and registers once per distinct task image (keyed by a digest
+of the build context) and every trial and skill arm of that image opens another
+session against the shared runtime. Concurrency is therefore bounded by
+sessions, not by how many images you have.
+
+Measured against the live service: 60 concurrent rollouts of one task ran on a
+single runtime in ~30s wall clock with no throttling and no failures (median
+session start ~15s). The session-creation rate quota (400/min) is the first
+thing you would hit, well before the 5,000 concurrent-session ceiling.
+
+Two quotas to check before a big run:
+
+| Quota | Default | Adjustable |
+|---|---|---|
+| Active Session Workloads per account | 5,000 | yes |
+| Total Agents (runtimes) per account | 100 | yes |
+| **Max image size** | **2 GB** | **no** |
+
+A full 88-task × 2-skill-arm matrix is 176 distinct images, over the default
+100-runtime cap — request an increase first. The 2 GB image limit is a hard
+service quota: heavy environments (LaTeX, Playwright, large model snapshots)
+cannot run here at all, and BenchFlow fails the build with a message naming the
+cap rather than letting it surface later as an opaque runtime error. Run those
+tasks on Docker or Daytona.
+
+Runtimes are shared, so they deliberately outlive the run that created them —
+like a built Docker image. Reclaim them by age:
+
+```bash
+bench sandbox cleanup --max-age 1440
+```
+
+Only runtimes tagged `benchflow-managed` are ever deleted, and cleanup is a
+no-op unless `BENCHFLOW_AGENTCORE_ROLE_ARN` is set.
+
 For large-scale runs (100+ tasks), use Daytona or Modal with high concurrency:
 
 ```bash

pyproject.toml

@@ -41,6 +41,9 @@ classifiers = [
 [project.optional-dependencies]
 dev = [
     "pre-commit>=3.7",
+    # AgentCore's CodeBuild path must reproduce Docker's ignore rules in tests
+    # even when the optional AWS backend is not installed.
+    "pathspec>=0.12",
     "pytest>=9.0.3",
     "pytest-asyncio>=0.24.0",
     "ruff>=0.7.0",
@@ -70,6 +73,17 @@ sandbox-modal = [
     "modal>=0.73",
     "tenacity>=8.0",
 ]
+sandbox-agentcore = [
+    # Bedrock AgentCore Runtime microVMs. The SDK supplies the interactive
+    # shell (`open_shell`) used as the ACP transport; boto3 supplies the
+    # control plane (CreateAgentRuntime) and the command plane
+    # (InvokeAgentRuntimeCommand). The boto3 floor is the first release that
+    # models the bedrock-agentcore/-control services.
+    "bedrock-agentcore>=1.18",
+    "boto3>=1.43.31",
+    # Docker-compatible build-context filtering without requiring a daemon.
+    "pathspec>=0.12",
+]
 bedrock = [
     "boto3>=1.40",
 ]

src/benchflow/acp/runtime.py

@@ -46,12 +46,6 @@
     build_priv_drop_cmd,
     enforce_agent_egress_firewall,
 )
-from benchflow.sandbox.process import (
-    AppleContainerProcess,
-    DaytonaProcess,
-    DaytonaPtyProcess,
-    DockerProcess,
-)
 from benchflow.trajectories._capture import _capture_session_trajectory
 
 # Re-exported for backwards compatibility — tests and downstream code
@@ -577,30 +571,13 @@ async def connect_acp(
             await asyncio.sleep(delay)
 
         try:
-            if environment == "docker":
-                live_proc = DockerProcess.from_sandbox_env(env)
-            elif environment == "apple-container":
-                live_proc = AppleContainerProcess.from_sandbox_env(env)
-            elif environment == "daytona":
-                transport_name = selected_acp_transport(
-                    agent=agent,
-                    environment=environment,
-                )
-                if transport_name == "ssh":
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
-                    logger.info("Using SSH transport for %s on Daytona", agent)
-                else:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for Daytona sandbox")
-            else:
-                is_dind = hasattr(env, "_strategy") and hasattr(
-                    env._strategy, "_compose_cmd"
-                )
-                if is_dind:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for DinD compose task")
-                else:
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
+            # The sandbox owns its transport: it knows how to reach into
+            # itself, so there is no provider-name branch here. Backends with
+            # no live-process transport raise an actionable NotImplementedError
+            # from BaseSandbox.live_process rather than silently receiving
+            # another backend's transport (Modal previously fell through to
+            # DaytonaProcess and failed deep inside SSH setup).
+            live_proc = await env.live_process(agent=agent)
 
             agent_log = rollout_dir / "agent" / f"{agent.replace('-', '_')}.txt"
             transport = ContainerTransport(

src/benchflow/cli/environment.py

@@ -145,7 +145,12 @@ def environment_cleanup(
             bool, typer.Option("--dry-run", help="List sandboxes without deleting")
         ] = False,
         max_age_minutes: Annotated[
-            int, typer.Option("--max-age", help="Delete sandboxes older than N minutes")
+            int,
+            typer.Option(
+                "--max-age",
+                min=0,
+                help="Delete sandboxes older than N minutes",
+            ),
         ] = 1440,
     ) -> None:
         """Deprecated; use `bench sandbox cleanup`."""

src/benchflow/cli/main.py

@@ -586,10 +586,11 @@ def eval_run(
         typer.Option("--trials", help="Number of trials for --matrix"),
     ] = 1,
 ) -> None:
-    """Run an evaluation — single task or batch.
-
-    Sandbox: docker, daytona, modal, or apple-container.
-    """
+    # The supported --sandbox values are rendered from the provider registry
+    # into that option's own help text. This docstring used to hand-copy them
+    # ("docker, daytona, modal, or apple-container") and went stale the moment
+    # a backend was added, so it no longer repeats them.
+    """Run an evaluation — single task or batch."""
     _apply_dotenv_to_process_env()
 
     request = EvalCreateRequest(

src/benchflow/cli/sandbox.py

@@ -124,24 +124,82 @@ def sandbox_list_local() -> None:
     console.print(f"\n[bold]{total} sandbox(es)[/bold]")
 
 
+def _cleanup_agentcore_runtimes(*, dry_run: bool, max_age_minutes: int) -> bool:
+    """Reap stale AgentCore runtimes. False when AgentCore is not in play.
+
+    AgentCore runtimes are shared across rollouts and intentionally outlive the
+    run that created them, so unlike Docker they accumulate — and *Total Agents
+    per Account* defaults to only 100.
+
+    Gated on ``BENCHFLOW_AGENTCORE_ROLE_ARN`` rather than merely on an
+    importable SDK. ``boto3`` arrives with several unrelated extras, so without
+    this gate ``bench sandbox cleanup`` would reach out to AWS on any machine
+    that happens to have credentials configured — including during tests.
+    """
+    import os
+
+    if not os.environ.get("BENCHFLOW_AGENTCORE_ROLE_ARN"):
+        return False
+    try:
+        import boto3
+
+        from benchflow.sandbox.agentcore_reaper import reap_stale_runtimes
+    except ImportError:
+        return False
+
+    region = os.environ.get("BENCHFLOW_AGENTCORE_REGION") or "us-west-2"
+    try:
+        control = boto3.Session(region_name=region).client("bedrock-agentcore-control")
+        report = reap_stale_runtimes(
+            control, max_age_minutes=max_age_minutes, dry_run=dry_run
+        )
+    except Exception as exc:
+        # Missing/expired credentials are a "nothing to do here", not a crash
+        # in the middle of a cleanup that may still have Daytona work to do.
+        print_error(f"Skipping AgentCore cleanup: {exc}")
+        return False
+
+    verb = "Would delete" if dry_run else "Deleted"
+    console.print(f"AgentCore ({region}): {report.summary()}")
+    if report.deleted:
+        console.print(f"  {verb}: {', '.join(report.deleted)}")
+    return True
+
+
 def sandbox_cleanup(*, dry_run: bool, max_age_minutes: int) -> None:
-    """Clean up orphaned Daytona sandboxes.
+    """Clean up orphaned Daytona sandboxes and stale AgentCore runtimes.
 
-    Like ``sandbox list``, this is a no-op (not an error) when the optional
-    Daytona SDK is absent: only Daytona has persistent sandboxes to reap.
+    Both are opt-in extras, so a missing SDK is a no-op rather than an error;
+    only these two backends leave anything behind between runs. Docker
+    sandboxes are torn down per run.
     """
-    if not _daytona_sdk_available():
+    cleaned_any = False
+
+    if _daytona_sdk_available():
+        from benchflow.cli import main as cli_main
+
+        try:
+            cli_main._cleanup_daytona_sandboxes(
+                dry_run=dry_run, max_age_minutes=max_age_minutes
+            )
+            cleaned_any = True
+        except (Exception, typer.Exit) as exc:
+            # The Daytona SDK is installed but unusable (typically no
+            # DAYTONA_API_KEY). Report and continue: a second backend may still
+            # have resources to reclaim, and aborting here would silently leave
+            # them behind — AgentCore runtimes consume a 100-per-account quota.
+            print_error(f"Skipping Daytona cleanup: {exc}")
+
+    if _cleanup_agentcore_runtimes(dry_run=dry_run, max_age_minutes=max_age_minutes):
+        cleaned_any = True
+
+    if not cleaned_any:
         console.print(
-            "Nothing to clean up. The Daytona SDK is not installed "
-            "([cyan]uv sync --extra sandbox-daytona[/cyan]); only Daytona has "
-            "persistent sandboxes to reap. Docker sandboxes are torn down per run."
+            "Nothing to clean up. Neither the Daytona SDK "
+            "([cyan]uv sync --extra sandbox-daytona[/cyan]) nor the AgentCore "
+            "extra ([cyan]uv sync --extra sandbox-agentcore[/cyan]) is "
+            "installed; only those backends leave resources between runs."
         )
-        return
-    from benchflow.cli import main as cli_main
-
-    cli_main._cleanup_daytona_sandboxes(
-        dry_run=dry_run, max_age_minutes=max_age_minutes
-    )
 
 
 def register_sandbox(app: typer.Typer) -> None:
@@ -173,8 +231,13 @@ def sandbox_cleanup_cmd(
             bool, typer.Option("--dry-run", help="List sandboxes without deleting")
         ] = False,
         max_age_minutes: Annotated[
-            int, typer.Option("--max-age", help="Delete sandboxes older than N minutes")
+            int,
+            typer.Option(
+                "--max-age",
+                min=0,
+                help="Delete sandboxes older than N minutes",
+            ),
         ] = 1440,
     ) -> None:
-        """Clean up orphaned Daytona sandboxes."""
+        """Clean up orphaned sandboxes and stale shared runtimes."""
         sandbox_cleanup(dry_run=dry_run, max_age_minutes=max_age_minutes)

src/benchflow/rollout/__init__.py

@@ -956,6 +956,13 @@ async def setup(self) -> None:
             self._timeout = int(cfg.timeout)
         else:
             self._timeout = int(self._task.config.agent.timeout_sec or 0)
+        # Look on the type, not via instance getattr: permissive proxies and
+        # AsyncMock-based test sandboxes manufacture arbitrary attributes,
+        # which would turn this optional synchronous hook into an un-awaited
+        # fake coroutine.
+        configure_timeout = getattr(type(self._env), "configure_agent_timeout", None)
+        if callable(configure_timeout):
+            configure_timeout(self._env, self._timeout)
 
         _write_config(
             self._rollout_dir,