feat(kubernetes): expose agent-sandbox operatingMode (suspend/resume) for idle scale-to-zero with PVC retention

[austin-shih]

Problem Statement

Disposable per-user sandboxes backed by durable PVCs (as proposed in open PR
#2034) give great continuity, but for lifecycle management Kubernetes OpenShell
exposes create/delete but no suspend/resume. There's no way to free a sandbox's**
**compute when idle while retaining its identity and PVC. For deployments with many
provisioned-but-often-idle users, keeping every pod running 24/7 is a large,
mostly-wasted compute cost; deleting on idle instead loses the sandbox identity
(and, without #2034, the data). We want: idle → free compute; next login → resume
with state intact.

This is the Kubernetes-driver realization of the general capability proposed in
#1823 (checkpoint/pause/resume), scoped concretely to what agent-sandbox already
implements.

Proposed Design

Surface agent-sandbox's existing suspend/resume capability through the OpenShell
Kubernetes driver and gateway/CLI — the lifecycle analog of how #2034 surfaced
pod-template/volume config through driver_config.kubernetes.

agent-sandbox already implements this in its Sandbox CRD and controller:

• v1beta1: spec.operatingMode: Running | Suspended (default Running).
• v1alpha1: the equivalent is spec.replicas (0 = suspended; the API
  conversion maps Suspended ↔ replicas=0).
• On Suspended, the controller deletes the backing Pod (frees CPU/memory)
  while leaving the Sandbox object and its PVCs in place — PVCs are reconciled
  independently and removed only when the Sandbox itself is deleted. Status surfaces
  a Suspended condition (PodTerminated / PodNotTerminated).
• On Running, the controller recreates the Pod and reattaches the same PVC(s).

What OpenShell would add:

• Driver: set operatingMode (v1beta1) / replicas=0 (v1alpha1) on the managed
  Sandbox CR to suspend, and flip back to resume.
• Gateway: keep the sandbox registered across a suspend (don't treat the absent
  Pod as a dead sandbox) and re-route on resume when the Pod returns.
• Interface: a lifecycle op (e.g. openshell sandbox suspend|resume) and/or an
  idle policy; resume triggered by the controlling app on session start.
• Existing seam: OpenShell already defines a StopSandbox RPC in the
  compute-driver contract (proto/compute_driver.proto), but it is currently
  unimplemented for the Kubernetes driver
  (crates/openshell-driver-kubernetes/src/grpc.rs) — a natural hook for wiring
  suspend, with resume as its counterpart.
• Pairs with open PR #2034: that PR proposes the durable, caller-owned per-user PVC;
  this gives the lifecycle to free its compute while keeping the data.

Alternatives Considered

• Always-on pods (status quo): simplest, but pays compute for every provisioned
  user, not just active ones — expensive at scale.
• Delete + recreate on idle/login: frees compute, but churns the sandbox identity
  and pays a full cold-create each login; with #2034 the data survives, but
  registration/orphan handling is messier than a first-class suspend.
• In-place pod restart only: doesn't free compute.
• → operatingMode suspend/resume is preferable: it's a first-class primitive
  already modeled and implemented in agent-sandbox; OpenShell only needs to expose
  and drive it.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

---

Related: #1823 (general checkpoint/pause/resume design), #1551 (VM-driver
suspend/resume). Builds on open PR #2034 by @mjamiv.

[mjamiv]

I reviewed the current implementation surface. My read is that this should get maintainer API review before a full code PR.

A narrow Kubernetes-driver patch for the existing internal StopSandbox RPC is possible: it could patch the agent-sandbox resource to spec.operatingMode: Suspended for v1beta1 or spec.replicas: 0 for v1alpha1. That would be a useful primitive, but it would not expose a user-facing suspend/resume flow by itself.

The full feature crosses several public control-plane decisions:

• public API shape for suspend and resume operations;
• whether SandboxPhase needs an explicit Suspended state;
• how watch/reconcile should treat an intentionally absent pod;
• how supervisor disconnects during suspend avoid being interpreted as provisioning/error;
• whether compute drivers need a new internal ResumeSandbox RPC, or whether resume is modeled some other way;
• CLI UX and idle-policy behavior.

I would split this after maintainer confirmation:

1. Driver primitive: implement Kubernetes StopSandbox as agent-sandbox suspend, with unit coverage for the v1beta1 and v1alpha1 patch shapes.
2. Public lifecycle API: add suspend/resume RPCs, CLI commands, phase semantics, and gateway reconciliation behavior.
3. E2E: verify suspend deletes the backing pod, PVCs remain, resume recreates a pod, and sandbox identity/routing recover correctly.

Happy to contribute the driver primitive once maintainers confirm the intended public lifecycle/API direction.

[mjamiv]

@elezar - Putting this item on your radar as it relates to #2034 and #2086

[TaylorMutch]

Suspend / resume overall is something I think we'd like to support. We have a number of other large changes in motion, but I think it would be ideal to have some consist expectation of how this might work across the current in-tree drivers rather than implemented for just Kubernetes.

[johntmyers]

@mjamiv could you give some rough metrics on pod volume, sandbox life duration, etc you are working with?

[mjamiv]

@johntmyers roughly, we are designing around one long-lived sandbox identity per user/agent, backed by durable PVC storage, with the pod treated as replaceable compute.

The long-run target is Kubernetes-scale operation for tens of thousands of sandboxes, with ~20k as the planning goal. That is why we want to move away from VM-style always-on pets toward durable identity/state plus fresh, disposable pods.

A few reasons this matters for us:

• We want pods regularly recreated from the latest image so runtime fixes, security updates, tool changes, and base config updates actually take effect.
• Users and agents can intentionally or accidentally mutate parts of the runtime filesystem that are hard to fully isolate. Replacing the pod gives us a clean image baseline again, while intended durable state stays on PVC.
• Usage is bursty, so idle pods should not have to consume compute indefinitely.

The main open design challenge is scheduled/cron-style work. Some agents need an "always-on" behavior for background jobs, but we still want the pod to stay fresh and be able to suspend/resume or rotate on a schedule. That probably needs an explicit lifecycle model rather than treating pod disappearance as failure.

So the target model is: durable sandbox identity/PVC, disposable pod compute, regular refresh from image, and a clear story for idle suspend/resume plus scheduled work.