feat(sdk/go): add SandboxPoolManager with fence/tombstone parity

docs/guides/client-pool.md

@@ -298,41 +298,71 @@ best-effort kill attempt. The Go method is intentionally outside the
 
 ### Retiring an old pool namespace
 
-The retirement procedure differs across SDKs because Go does not currently ship a
-`SandboxPoolManager` or the destroy / tombstone primitives that Python and Kotlin
-have.
-
-**Python / Kotlin** — use `SandboxPoolManager.destroy(poolName, options)`. The manager
-applies a full `DESTROYING → DESTROYED` protocol: write a `DESTROYING` fence into the
-state store (so any still-running peer instance sees it), best-effort drain and kill
-every idle sandbox up to `drain_timeout`, clear the persistent per-pool state, then
-write a `DESTROYED` tombstone with `tombstone_ttl` (default 7 days) so future callers
-cannot silently rebind to the same `pool_name`.
-
-**Go** — the Go SDK has no equivalent API and no state-store primitives for
-tombstones or fences. The closest safe approximation is an operator-driven, out-of-band
-sequence:
-
-1. Stop every process that instantiates a pool against the old `pool_name`. Call
-   `pool.Shutdown(ctx, true)` on each. This releases each node's primary lock but
-   leaves idle entries in the store.
-2. From one still-alive pool instance (or a throwaway one bound to the same
-   `PoolName` + `StateStore`), call `pool.ReleaseAllIdle(ctx)` to drain and
-   best-effort kill every idle sandbox.
-3. Set `store.SetMaxIdle(ctx, poolName, 0)` so any peer that races back in cannot
-   warm up new sandboxes.
-4. Move all future callers to a new `PoolName` (for example
-   `orders-v2` → `orders-v3`). This is the Go substitute for the `DESTROYED`
-   tombstone: without a shared marker, name rotation is the only way to guarantee no
-   accidental reuse.
-5. If you are using the Redis store and want to reclaim keys, delete them directly
-   with `DEL` / `SCAN` against your Redis instance — the Go SDK does not expose a
-   destroy helper for this.
-
-Without a fence, steps 2 and 3 race with any surviving peer that has not yet been
-stopped. If you cannot guarantee "all writers stopped" before step 2, the only correct
-option is to rotate `PoolName` first (step 4) and let the old namespace's idle entries
-expire naturally via `idle_timeout`.
+Every SDK exposes a `SandboxPoolManager` with a `destroy` operation that applies the
+same `DESTROYING → DESTROYED` protocol:
+
+1. Write a `DESTROYING` fence into the state store, so any still-running peer instance
+   sees it and stops replenishing instead of racing the retirement.
+2. Best-effort drain and kill every idle sandbox, bounded by the drain timeout.
+3. Clear the persistent per-pool state.
+4. Write a `DESTROYED` tombstone with the tombstone TTL (default 7 days) so future
+   callers cannot silently rebind to the same `pool_name`.
+
+Destroy is idempotent: calling it on an already-tombstoned namespace reports
+`DESTROYED` without draining or killing anything. If the drain or the cleanup cannot
+finish, the namespace stays `DESTROYING` and the call reports the destroy as
+incomplete; retrying is safe and picks up where it left off.
+
+**Python / Kotlin** — `SandboxPoolManager.destroy(poolName, options)`, configured
+through `PoolDestroyOptions` (`strategy`, `drain_timeout`, `tombstone_ttl`).
+
+**Go** — `(*SandboxPoolManager).Destroy(ctx, poolName, options)`:
+
+```go
+manager, err := opensandbox.NewSandboxPoolManagerBuilder().
+    StateStore(store).
+    ConnectionConfig(connCfg).
+    Build()
+if err != nil {
+    return err
+}
+
+result, err := manager.Destroy(ctx, "orders-v2", opensandbox.PoolDestroyOptions{})
+if err != nil {
+    return err
+}
+log.Printf("retired %s: drained=%d killed=%d",
+    result.PoolName, result.DrainedIdleCount, result.KilledIdleCount)
+```
+
+`PoolDestroyOptions` mirrors the other SDKs. `Strategy` selects the algorithm and only
+`PoolDestroyForce` is implemented. `DrainTimeout` and `TombstoneTTL` are `*time.Duration`:
+leave them nil for the defaults (30s and 7 days), or set an explicit zero to drain
+without a deadline and to write a tombstone that never expires.
+
+The fence is what makes retirement safe without stopping every writer first, and it
+is enforced on two levels. The state store refuses `PutIdle`, `SetMaxIdle` and
+`SetIdleEntryTTL` with a `*PoolDestroyedError` and hands out no primary lock, which
+stops replenishment. The pool itself also checks the fence when it starts, before every
+acquire, again once an acquire holds a live sandbox, and on each reconcile tick: a
+surviving peer stops outright on its next tick, an in-flight acquire fails rather
+than minting a fresh sandbox into the retired namespace through the direct-create
+fallthrough, and a sandbox obtained just before the fence landed is killed instead
+of handed out. The post-acquire check matters because the idle take is deliberately
+left unfenced so `destroy` can drain: once an ID has been taken, `destroy` can no
+longer reach it, so the acquire has to dispose of it itself.
+Starting a fresh pool against a tombstoned `PoolName` fails for the same reason, so
+rebinding the name requires either waiting out the tombstone TTL or rotating to a new
+`PoolName`.
+
+One deliberate exception: if the state store itself is unreachable, the destroy state
+is unknowable, so policies that already fall through to direct create on a store
+outage (`DIRECT_CREATE`, `RETRY_NEXT_IDLE_THEN_CREATE`) assume `ACTIVE` and proceed,
+matching the existing `try_take_idle` outage behavior in the OSEP-0005 error-code
+matrix. `FAIL_FAST` and `RETRY_NEXT_IDLE` surface the outage instead. That relaxation
+stops at a sandbox already taken from the idle buffer: there the check is fail-closed
+and an unreachable store means the sandbox is killed, because nothing else is tracking
+it any more.
 
 ## Further reading
 

sdks/sandbox/go/pool.go

@@ -16,6 +16,7 @@ package opensandbox
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"sync"
 	"sync/atomic"
@@ -91,6 +92,20 @@ func (p *DefaultSandboxPool) Start(ctx context.Context) error {
 	startMaxIdle := p.config.MaxIdle
 	p.mu.Unlock()
 
+	// Refuse to bind a retired namespace. Only a definite fence blocks startup;
+	// a store outage is left to the writes below to surface.
+	if err := p.ensureNamespaceActive(ctx); err != nil {
+		var destroyed *PoolDestroyedError
+		if errors.As(err, &destroyed) {
+			p.mu.Lock()
+			if p.lifecycleState == PoolLifecycleStarting {
+				p.lifecycleState = PoolLifecycleNotStarted
+			}
+			p.mu.Unlock()
+			return err
+		}
+	}
+
 	// Initialize state store with pool configuration.
 	if err := p.config.StateStore.SetMaxIdle(ctx, p.config.PoolName, startMaxIdle); err != nil {
 		p.mu.Lock()
@@ -187,6 +202,17 @@ func (p *DefaultSandboxPool) syncHealthState() {
 func (p *DefaultSandboxPool) runReconcileTick(ctx context.Context) {
 	p.reconMu.Lock()
 	defer p.reconMu.Unlock()
+
+	// A destroy fences the namespace for every peer. Stop rather than keep
+	// replenishing a pool that is being retired.
+	if err := p.ensureNamespaceActive(ctx); err != nil {
+		var destroyed *PoolDestroyedError
+		if errors.As(err, &destroyed) {
+			p.stopAfterNamespaceDestroyed(destroyed.State)
+			return
+		}
+	}
+
 	createFn := func(ctx context.Context, reason PooledSandboxCreateReason) (string, error) {
 		return p.createOneSandbox(ctx, reason)
 	}
@@ -215,6 +241,12 @@ func (p *DefaultSandboxPool) Acquire(ctx context.Context, opts AcquireOptions) (
 		policy = *opts.Policy
 	}
 
+	// A fenced namespace must not mint new sandboxes, so this has to run before the
+	// direct-create fallthrough below and not only on the store write paths.
+	if err := p.ensureNamespaceActiveForAcquire(ctx, policy); err != nil {
+		return nil, err
+	}
+
 	// Resolve minTTL.
 	minTTL := p.config.AcquireMinRemainingTTL
 	if opts.MinRemainingTTL > 0 {
@@ -297,6 +329,12 @@ func (p *DefaultSandboxPool) Acquire(ctx context.Context, opts AcquireOptions) (
 				go p.killDiscardedAliveSandboxes(pendingKill)
 				return nil, &PoolNotRunningError{PoolName: p.config.PoolName, State: currentState}
 			}
+			// A destroy may have landed since the preflight check. Stop retrying rather
+			// than pop further idle IDs out from under the drain.
+			if err := p.ensureNamespaceActiveForAcquire(ctx, policy); err != nil {
+				go p.killDiscardedAliveSandboxes(pendingKill)
+				return nil, err
+			}
 			continue
 		}
 		// Connect + readiness succeeded. From here on the sandbox is a healthy, borrowable idle:
@@ -320,6 +358,15 @@ func (p *DefaultSandboxPool) Acquire(ctx context.Context, opts AcquireOptions) (
 				return nil, fmt.Errorf("opensandbox: pool acquire: renew after connect failed: %w", renewErr)
 			}
 		}
+		// TryTakeIdle is unfenced so the destroy manager can drain, so this ID is
+		// already out of the store and a destroy can no longer reach it. Re-check
+		// before handing it over, fail-closed: if the store cannot confirm the
+		// namespace is ACTIVE, kill the sandbox rather than leak it into a
+		// namespace that may be retired.
+		if err := p.ensureNamespaceActiveAfterCreate(ctx, sb, nil); err != nil {
+			go p.killDiscardedAliveSandboxes(pendingKill)
+			return nil, err
+		}
 		go p.killDiscardedAliveSandboxes(pendingKill)
 		p.config.Logger.Debug("acquire: from idle",
 			"pool_name", p.config.PoolName,
@@ -348,7 +395,7 @@ func (p *DefaultSandboxPool) Acquire(ctx context.Context, opts AcquireOptions) (
 		"attempted_any", attemptedAny,
 		"loop_exhausted", loopExhausted,
 		"last_sandbox_id", lastSandboxID)
-	return p.directCreate(ctx, opts)
+	return p.directCreate(ctx, opts, policy)
 }
 
 // tryTakeIdle wraps the store's take primitives, returning a nil result on a legitimate empty
@@ -404,7 +451,7 @@ func (p *DefaultSandboxPool) connectIdle(ctx context.Context, sandboxID string,
 	})
 }
 
-func (p *DefaultSandboxPool) directCreate(ctx context.Context, opts AcquireOptions) (*Sandbox, error) {
+func (p *DefaultSandboxPool) directCreate(ctx context.Context, opts AcquireOptions, policy AcquirePolicy) (*Sandbox, error) {
 	var sb *Sandbox
 	var err error
 
@@ -428,7 +475,15 @@ func (p *DefaultSandboxPool) directCreate(ctx context.Context, opts AcquireOptio
 	if err != nil {
 		return nil, err
 	}
-	return p.postCreateChecks(ctx, sb, opts)
+	sb, err = p.postCreateChecks(ctx, sb, opts)
+	if err != nil {
+		return nil, err
+	}
+	// Re-check: a destroy may have landed while this sandbox was being created.
+	if err := p.ensureNamespaceActiveAfterCreate(ctx, sb, &policy); err != nil {
+		return nil, err
+	}
+	return sb, nil
 }
 
 // postCreateChecks applies renew to a freshly created sandbox.
@@ -810,6 +865,109 @@ done:
 	return nil
 }
 
+// ensureNamespaceActive returns *PoolDestroyedError when a destroy has fenced or
+// tombstoned this pool's namespace, and *PoolStateStoreUnavailableError when the
+// state store cannot answer.
+func (p *DefaultSandboxPool) ensureNamespaceActive(ctx context.Context) error {
+	state, err := p.config.StateStore.GetDestroyState(ctx, p.config.PoolName)
+	if err != nil {
+		var unavailable *PoolStateStoreUnavailableError
+		if errors.As(err, &unavailable) {
+			return err
+		}
+		return &PoolStateStoreUnavailableError{Operation: "GetDestroyState", Cause: err}
+	}
+	if state != PoolDestroyStateActive {
+		return &PoolDestroyedError{PoolName: p.config.PoolName, State: state}
+	}
+	return nil
+}
+
+// ensureNamespaceActiveForAcquire is ensureNamespaceActive with the same
+// store-outage degradation the take path already applies: policies that fall
+// through to direct create treat an unreachable store as "state unknown" and
+// proceed, so a full store outage does not make them less available than
+// documented (OSEP-0005 error-code matrix). Fail-closed policies surface it.
+func (p *DefaultSandboxPool) ensureNamespaceActiveForAcquire(ctx context.Context, policy AcquirePolicy) error {
+	err := p.ensureNamespaceActive(ctx)
+	if err == nil {
+		return nil
+	}
+	var unavailable *PoolStateStoreUnavailableError
+	if errors.As(err, &unavailable) && policyFallsThroughToDirectCreate(policy) {
+		p.config.Logger.Warn("acquire: state store unavailable during namespace check, "+
+			"assuming ACTIVE and degrading to direct create",
+			"pool_name", p.config.PoolName,
+			"policy", policy,
+			"error", err)
+		return nil
+	}
+	return err
+}
+
+// ensureNamespaceActiveAfterCreate re-checks the fence once the acquire path holds
+// a live sandbox, so a destroy that landed mid-acquire does not leak one into a
+// retired namespace. On a fence the sandbox is killed and closed.
+//
+// policy is non-nil only for the direct-create path, where a store outage degrades
+// the same way the rest of that path does. The idle path passes nil and stays
+// fail-closed: that sandbox is already out of the store, so an unconfirmed
+// namespace has to be treated as retired.
+func (p *DefaultSandboxPool) ensureNamespaceActiveAfterCreate(ctx context.Context, sb *Sandbox, policy *AcquirePolicy) error {
+	err := p.ensureNamespaceActive(ctx)
+	if err == nil {
+		return nil
+	}
+	var unavailable *PoolStateStoreUnavailableError
+	if errors.As(err, &unavailable) && policy != nil && policyFallsThroughToDirectCreate(*policy) {
+		p.config.Logger.Warn("acquire: state store unavailable during post-create namespace check, "+
+			"keeping sandbox and degrading per policy",
+			"pool_name", p.config.PoolName,
+			"sandbox_id", sb.ID(),
+			"policy", *policy,
+			"error", err)
+		return nil
+	}
+	go p.killSandboxBestEffort(sb.ID())
+	_ = sb.Close()
+	return err
+}
+
+// stopAfterNamespaceDestroyed stops the pool once its namespace has been retired.
+// It runs on the reconcile goroutine, so unlike Shutdown it must not wait on p.wg.
+func (p *DefaultSandboxPool) stopAfterNamespaceDestroyed(state PoolDestroyState) {
+	p.mu.Lock()
+	if p.lifecycleState == PoolLifecycleStopped || p.lifecycleState == PoolLifecycleDraining {
+		p.mu.Unlock()
+		return
+	}
+	p.lifecycleState = PoolLifecycleStopped
+	if p.ticker != nil {
+		p.ticker.Stop()
+	}
+	if p.done != nil && !p.doneClosed {
+		close(p.done)
+		p.doneClosed = true
+	}
+	cancelFn := p.reconCancel
+	sdCh := p.shutdownDone
+	p.mu.Unlock()
+
+	if cancelFn != nil {
+		cancelFn()
+	}
+	if sdCh != nil {
+		select {
+		case <-sdCh:
+		default:
+			close(sdCh)
+		}
+	}
+	p.config.Logger.Info("pool stopped: namespace destroyed",
+		"pool_name", p.config.PoolName,
+		"destroy_state", state)
+}
+
 const (
 	killSandboxTimeout = 30 * time.Second
 )

sdks/sandbox/go/pool_errors.go

@@ -63,3 +63,31 @@ func (e *PoolStateStoreUnavailableError) Error() string {
 }
 
 func (e *PoolStateStoreUnavailableError) Unwrap() error { return e.Cause }
+
+// PoolDestroyedError is returned when a write targets a pool namespace that a
+// destroy has fenced, i.e. one that is DESTROYING or DESTROYED.
+type PoolDestroyedError struct {
+	PoolName string
+	State    PoolDestroyState
+}
+
+func (e *PoolDestroyedError) Error() string {
+	return fmt.Sprintf("opensandbox: pool %q is %s", e.PoolName, e.State)
+}
+
+// PoolDestroyIncompleteError is returned when a destroy could not run to
+// completion. The namespace stays DESTROYING and the caller should retry.
+type PoolDestroyIncompleteError struct {
+	PoolName string
+	Reason   string
+	Cause    error
+}
+
+func (e *PoolDestroyIncompleteError) Error() string {
+	if e.Cause == nil {
+		return fmt.Sprintf("opensandbox: pool %q destroy incomplete: %s", e.PoolName, e.Reason)
+	}
+	return fmt.Sprintf("opensandbox: pool %q destroy incomplete: %s: %v", e.PoolName, e.Reason, e.Cause)
+}
+
+func (e *PoolDestroyIncompleteError) Unwrap() error { return e.Cause }