feat(sandbox): add Amazon Bedrock AgentCore Runtime backend

docs/running-benchmarks.md

@@ -347,6 +347,7 @@ The **Harvey LAB harness** agent is special — it runs Harvey LAB's own agent l
 | Apple Container | `--sandbox apple-container` | Local Apple Silicon macOS runs without Docker Desktop |
 | Daytona | `--sandbox daytona` | Cloud runs with concurrency (needs `DAYTONA_API_KEY`) |
 | Modal | `--sandbox modal` | Serverless, high concurrency (needs Modal auth) |
+| AgentCore | `--sandbox agentcore` | AWS-native isolated microVMs (needs AWS credentials) |
 
 Apple Container requires Apple Container 1.1+ on Apple Silicon and runs the model
 proxy inside each VM. It supports public-network, single-container arm64 tasks and
@@ -356,6 +357,76 @@ running concurrent BenchFlow processes, because the macOS allocation leak is
 system-wide. Use Docker, Daytona, or Modal for `network_mode = "no-network"`,
 multi-service, snapshot, or high-concurrency runs.
 
+### Amazon Bedrock AgentCore
+
+`--sandbox agentcore` runs each rollout in an AgentCore Runtime microVM.
+Install the extra and point BenchFlow at an execution role:
+
+```bash
+pip install 'benchflow[sandbox-agentcore]'
+export BENCHFLOW_AGENTCORE_ROLE_ARN="arn:aws:iam::<account>:role/<role>"
+export BENCHFLOW_AGENTCORE_REGION="us-west-2"   # optional, this is the default
+```
+
+The role must be assumable by `bedrock-agentcore.amazonaws.com` and able to
+pull from ECR and write CloudWatch logs. Your own credentials additionally need
+`bedrock-agentcore:*` and ECR push permissions.
+
+Unlike the other backends, AgentCore cannot take a container image directly:
+BenchFlow builds the task image for `linux/arm64`, pushes it to ECR (repository
+`benchflow-agentcore`, override with `BENCHFLOW_AGENTCORE_ECR_REPOSITORY`), and
+registers it as an agent runtime. The image is tagged by content digest, so a
+task whose environment has not changed reuses the pushed layers and the
+existing runtime instead of paying for a rebuild.
+
+BenchFlow also appends a small HTTP responder to the image and makes it the
+entrypoint. AgentCore refuses command execution for a session whose container
+does not answer `GET /ping` on port 8080, and task images know nothing about
+AgentCore, so this shim is what makes an ordinary task image runnable.
+
+Constraints: `linux/arm64` only, single container (no compose/multi-service
+tasks), no snapshot support, and `network_mode = "no-network"` is **not**
+enforceable — AgentCore's network mode is either `PUBLIC` or `VPC`, so
+BenchFlow refuses no-network tasks on this backend rather than running them
+unisolated. The model proxy runs inside the sandbox, as on Daytona and Modal.
+Sessions default to a 15-minute idle timeout and an 8-hour lifetime; override
+with `BENCHFLOW_AGENTCORE_IDLE_TIMEOUT_SEC` / `BENCHFLOW_AGENTCORE_MAX_LIFETIME_SEC`
+if agent turns are long enough to risk reclamation mid-run.
+
+**Running a large matrix.** A rollout is a *session*, not a runtime: one
+registered runtime hosts many concurrent, filesystem-isolated microVMs. The
+account quotas point the same way — 5,000 concurrent *Active Session Workloads*
+against only 100 *Total Agents*, with `CreateAgentRuntime` limited to 5/s. So
+BenchFlow builds and registers once per distinct task image (keyed by a digest
+of the build context) and every trial and skill arm of that image opens another
+session against the shared runtime. Concurrency is therefore bounded by
+sessions, not by how many images you have.
+
+Two quotas to check before a big run:
+
+| Quota | Default | Adjustable |
+|---|---|---|
+| Active Session Workloads per account | 5,000 | yes |
+| Total Agents (runtimes) per account | 100 | yes |
+| **Max image size** | **2 GB** | **no** |
+
+A full 88-task × 2-skill-arm matrix is 176 distinct images, over the default
+100-runtime cap — request an increase first. The 2 GB image limit is a hard
+service quota: heavy environments (LaTeX, Playwright, large model snapshots)
+cannot run here at all, and BenchFlow fails the build with a message naming the
+cap rather than letting it surface later as an opaque runtime error. Run those
+tasks on Docker or Daytona.
+
+Runtimes are shared, so they deliberately outlive the run that created them —
+like a built Docker image. Reclaim them by age:
+
+```bash
+bench sandbox cleanup --max-age 1440
+```
+
+Only runtimes tagged `benchflow-managed` are ever deleted, and cleanup is a
+no-op unless `BENCHFLOW_AGENTCORE_ROLE_ARN` is set.
+
 For large-scale runs (100+ tasks), use Daytona or Modal with high concurrency:
 
 ```bash

pyproject.toml

@@ -70,6 +70,15 @@ sandbox-modal = [
     "modal>=0.73",
     "tenacity>=8.0",
 ]
+sandbox-agentcore = [
+    # Bedrock AgentCore Runtime microVMs. The SDK supplies the interactive
+    # shell (`open_shell`) used as the ACP transport; boto3 supplies the
+    # control plane (CreateAgentRuntime) and the command plane
+    # (InvokeAgentRuntimeCommand). The boto3 floor is the first release that
+    # models the bedrock-agentcore/-control services.
+    "bedrock-agentcore>=1.18",
+    "boto3>=1.43.31",
+]
 bedrock = [
     "boto3>=1.40",
 ]

src/benchflow/acp/runtime.py

@@ -46,12 +46,6 @@
     build_priv_drop_cmd,
     enforce_agent_egress_firewall,
 )
-from benchflow.sandbox.process import (
-    AppleContainerProcess,
-    DaytonaProcess,
-    DaytonaPtyProcess,
-    DockerProcess,
-)
 from benchflow.trajectories._capture import _capture_session_trajectory
 
 # Re-exported for backwards compatibility — tests and downstream code
@@ -577,30 +571,13 @@ async def connect_acp(
             await asyncio.sleep(delay)
 
         try:
-            if environment == "docker":
-                live_proc = DockerProcess.from_sandbox_env(env)
-            elif environment == "apple-container":
-                live_proc = AppleContainerProcess.from_sandbox_env(env)
-            elif environment == "daytona":
-                transport_name = selected_acp_transport(
-                    agent=agent,
-                    environment=environment,
-                )
-                if transport_name == "ssh":
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
-                    logger.info("Using SSH transport for %s on Daytona", agent)
-                else:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for Daytona sandbox")
-            else:
-                is_dind = hasattr(env, "_strategy") and hasattr(
-                    env._strategy, "_compose_cmd"
-                )
-                if is_dind:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for DinD compose task")
-                else:
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
+            # The sandbox owns its transport: it knows how to reach into
+            # itself, so there is no provider-name branch here. Backends with
+            # no live-process transport raise an actionable NotImplementedError
+            # from BaseSandbox.live_process rather than silently receiving
+            # another backend's transport (Modal previously fell through to
+            # DaytonaProcess and failed deep inside SSH setup).
+            live_proc = await env.live_process(agent=agent)
 
             agent_log = rollout_dir / "agent" / f"{agent.replace('-', '_')}.txt"
             transport = ContainerTransport(

src/benchflow/cli/main.py

@@ -586,10 +586,11 @@ def eval_run(
         typer.Option("--trials", help="Number of trials for --matrix"),
     ] = 1,
 ) -> None:
-    """Run an evaluation — single task or batch.
-
-    Sandbox: docker, daytona, modal, or apple-container.
-    """
+    # The supported --sandbox values are rendered from the provider registry
+    # into that option's own help text. This docstring used to hand-copy them
+    # ("docker, daytona, modal, or apple-container") and went stale the moment
+    # a backend was added, so it no longer repeats them.
+    """Run an evaluation — single task or batch."""
     _apply_dotenv_to_process_env()
 
     request = EvalCreateRequest(

src/benchflow/cli/sandbox.py

@@ -124,24 +124,82 @@ def sandbox_list_local() -> None:
     console.print(f"\n[bold]{total} sandbox(es)[/bold]")
 
 
+def _cleanup_agentcore_runtimes(*, dry_run: bool, max_age_minutes: int) -> bool:
+    """Reap stale AgentCore runtimes. False when AgentCore is not in play.
+
+    AgentCore runtimes are shared across rollouts and intentionally outlive the
+    run that created them, so unlike Docker they accumulate — and *Total Agents
+    per Account* defaults to only 100.
+
+    Gated on ``BENCHFLOW_AGENTCORE_ROLE_ARN`` rather than merely on an
+    importable SDK. ``boto3`` arrives with several unrelated extras, so without
+    this gate ``bench sandbox cleanup`` would reach out to AWS on any machine
+    that happens to have credentials configured — including during tests.
+    """
+    import os
+
+    if not os.environ.get("BENCHFLOW_AGENTCORE_ROLE_ARN"):
+        return False
+    try:
+        import boto3
+
+        from benchflow.sandbox.agentcore_reaper import reap_stale_runtimes
+    except ImportError:
+        return False
+
+    region = os.environ.get("BENCHFLOW_AGENTCORE_REGION") or "us-west-2"
+    try:
+        control = boto3.Session(region_name=region).client("bedrock-agentcore-control")
+        report = reap_stale_runtimes(
+            control, max_age_minutes=max_age_minutes, dry_run=dry_run
+        )
+    except Exception as exc:
+        # Missing/expired credentials are a "nothing to do here", not a crash
+        # in the middle of a cleanup that may still have Daytona work to do.
+        print_error(f"Skipping AgentCore cleanup: {exc}")
+        return False
+
+    verb = "Would delete" if dry_run else "Deleted"
+    console.print(f"AgentCore ({region}): {report.summary()}")
+    if report.deleted:
+        console.print(f"  {verb}: {', '.join(report.deleted)}")
+    return True
+
+
 def sandbox_cleanup(*, dry_run: bool, max_age_minutes: int) -> None:
-    """Clean up orphaned Daytona sandboxes.
+    """Clean up orphaned Daytona sandboxes and stale AgentCore runtimes.
 
-    Like ``sandbox list``, this is a no-op (not an error) when the optional
-    Daytona SDK is absent: only Daytona has persistent sandboxes to reap.
+    Both are opt-in extras, so a missing SDK is a no-op rather than an error;
+    only these two backends leave anything behind between runs. Docker
+    sandboxes are torn down per run.
     """
-    if not _daytona_sdk_available():
+    cleaned_any = False
+
+    if _daytona_sdk_available():
+        from benchflow.cli import main as cli_main
+
+        try:
+            cli_main._cleanup_daytona_sandboxes(
+                dry_run=dry_run, max_age_minutes=max_age_minutes
+            )
+            cleaned_any = True
+        except (Exception, typer.Exit) as exc:
+            # The Daytona SDK is installed but unusable (typically no
+            # DAYTONA_API_KEY). Report and continue: a second backend may still
+            # have resources to reclaim, and aborting here would silently leave
+            # them behind — AgentCore runtimes consume a 100-per-account quota.
+            print_error(f"Skipping Daytona cleanup: {exc}")
+
+    if _cleanup_agentcore_runtimes(dry_run=dry_run, max_age_minutes=max_age_minutes):
+        cleaned_any = True
+
+    if not cleaned_any:
         console.print(
-            "Nothing to clean up. The Daytona SDK is not installed "
-            "([cyan]uv sync --extra sandbox-daytona[/cyan]); only Daytona has "
-            "persistent sandboxes to reap. Docker sandboxes are torn down per run."
+            "Nothing to clean up. Neither the Daytona SDK "
+            "([cyan]uv sync --extra sandbox-daytona[/cyan]) nor the AgentCore "
+            "extra ([cyan]uv sync --extra sandbox-agentcore[/cyan]) is "
+            "installed; only those backends leave resources between runs."
         )
-        return
-    from benchflow.cli import main as cli_main
-
-    cli_main._cleanup_daytona_sandboxes(
-        dry_run=dry_run, max_age_minutes=max_age_minutes
-    )
 
 
 def register_sandbox(app: typer.Typer) -> None:

src/benchflow/sandbox/_base.py

@@ -14,7 +14,7 @@
 import shlex
 from abc import ABC, abstractmethod
 from pathlib import Path
-from typing import Any
+from typing import TYPE_CHECKING, Any
 from uuid import uuid4
 
 from pydantic import BaseModel
@@ -24,6 +24,9 @@
 from benchflow.task.env import resolve_env_vars
 from benchflow.task.paths import RolloutPaths
 
+if TYPE_CHECKING:
+    from benchflow.sandbox.process import LiveProcess
+
 logger = logging.getLogger("benchflow")
 
 # Docker Compose service names are restricted to this grammar. Used to filter
@@ -386,6 +389,28 @@ async def is_file(
     async def attach(self) -> None:
         raise NotImplementedError("This environment does not support attaching.")
 
+    async def live_process(self, *, agent: str | None = None) -> LiveProcess:
+        """Open the bidirectional line pipe an ACP agent speaks over.
+
+        ``agent`` names the agent being launched; backends that pick a
+        transport per agent (Daytona: SSH for gemini, PTY otherwise) use it.
+
+        Each backend knows how to reach into its own sandbox, so the transport
+        choice belongs here rather than in a provider-name ``if/elif`` at the
+        ACP layer — that chain had to re-derive the backend from a string while
+        already holding this object, and sniff private attributes
+        (``env._strategy``) to tell Daytona's variants apart.
+
+        Backends with no live-process transport (Modal) leave this default in
+        place and fail with an actionable message instead of being handed
+        another backend's transport.
+        """
+        raise NotImplementedError(
+            f"{type(self).__name__} does not provide a live agent transport, so "
+            "it cannot host an ACP agent. Use a backend that implements "
+            "live_process() (docker, daytona, apple-container, agentcore)."
+        )
+
     # Container-level snapshot/restore (Branch substrate)
     #
     # Part of the Sandbox contract (``docs/architecture.md``): the Branch