feat(server): measure lifecycle operations server-side

docs/guides/sdk-telemetry.md

@@ -45,7 +45,35 @@ After create succeeds or fails, the SDK fire-and-forget posts to `POST /v1/metri
 - `sandboxId` / `image` may be omitted when create fails early.
 - SDK language and version come from the HTTP `User-Agent` header (for example `OpenSandbox-Python-SDK/0.1.15`), not from body fields.
 
-The server accepts the event with `204` and, when `[otel]` is enabled, records an OTEL histogram. See [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
+The server accepts the event with `204` and, when `[otel]` is enabled, records an OTEL histogram (`opensandbox.sandbox.create.duration`). See [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
+
+## SDK-reported vs server-measured
+
+This page describes what the **SDK** reports. `[otel]` also exports metrics the **server**
+measures itself, and the difference matters when you build dashboards:
+
+| Metric | Measured by | Covers |
+|---|---|---|
+| `opensandbox.sandbox.create.duration` | the SDK | the client's whole experience, network and SDK overhead included |
+| `opensandbox.sandbox.operation.duration` | the server | the server's own work at the API boundary |
+| `opensandbox.sandbox.operation.total` | the server | the same, as a count, with the error code on failure |
+
+Two consequences:
+
+- The SDK histogram **only exists where clients report it**. An integration calling the REST
+  API directly produces none of it, so a dashboard built on it alone will look empty rather
+  than healthy. The `operation.*` pair exists in every deployment.
+- `operation.total` carries `error.code` from the server's own taxonomy, which is what makes
+  failures queryable:
+  `sum by (error_code) (rate(opensandbox_sandbox_operation_total{outcome="error"}[5m]))`.
+
+Both cover provisioning: despite the `202 Accepted` status, `POST /sandboxes` blocks until
+the sandbox exists — Kubernetes awaits pod readiness, Docker awaits container start and
+egress sidecar readiness. So `operation.duration{operation="create"}` is a **server-side
+cold-start signal**, and unlike the SDK histogram it does not depend on clients reporting
+anything. The difference between the two is the client's share: network, auth, SDK overhead.
+
+Full attribute reference: [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
 
 ## When it runs
 

server/configuration.md

@@ -300,7 +300,7 @@ Per-sandbox enablement uses create request extensions (see OSEP-0009 and `exampl
 
 ## `[otel]`
 
-Optional OpenTelemetry metrics export for SDK-reported sandbox creation latency (`POST /v1/metrics/events`). Off by default; the ingestion endpoint still accepts events and records them as noop.
+Optional OpenTelemetry metrics export. Off by default; the ingestion endpoint still accepts events and records them as noop.
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
@@ -309,6 +309,41 @@ Optional OpenTelemetry metrics export for SDK-reported sandbox creation latency
 | `service_name` | string | `"opensandbox-server"` | `service.name` resource attribute. |
 | `export_interval_millis` | integer | `60000` | Periodic export interval (≥ 1000). |
 
+### Exported metrics
+
+| Metric | Type | Unit | Source |
+|---|---|---|---|
+| `opensandbox.sandbox.create.duration` | Histogram | `ms` | **Reported by the SDK** via `POST /v1/metrics/events` |
+| `opensandbox.sandbox.operation.duration` | Histogram | `ms` | Measured by the server at the API boundary |
+| `opensandbox.sandbox.operation.total` | Counter | - | Measured by the server at the API boundary |
+
+The distinction matters. `create.duration` is a number the client measured and sent, so it
+covers the client's whole experience — network and SDK overhead included — and **only exists
+in deployments whose clients report it**. An integration that calls the REST API directly
+produces none of it.
+
+The `operation.*` pair is the server timing its own work, so it exists in every deployment.
+Attributes:
+
+- `operation` — a lifecycle verb from a closed set: `create`, `update_metadata`, `delete`,
+  `pause`, `resume`, `renew`, `create_snapshot`, `delete_snapshot`. Read-only endpoints are
+  not instrumented.
+- `outcome` — `success` or `error`.
+- `error.code` — on the counter only, and only when the operation failed. Values come from
+  `SandboxErrorCodes` (`KUBERNETES::POD_READY_TIMEOUT`, `DOCKER::SANDBOX_IMAGE_PULL_FAILED`,
+  and so on), so the server's existing error taxonomy becomes queryable:
+  `sum by (error_code) (rate(opensandbox_sandbox_operation_total{outcome="error"}[5m]))`.
+
+`create` covers **provisioning, not just scheduling**, despite the `202 Accepted` status: the
+handler blocks until the sandbox exists. In Kubernetes `create_sandbox` awaits
+`_wait_for_sandbox_ready`, so the sample includes pod readiness; in Docker it awaits the
+provisioning thread, so it includes container start and egress sidecar readiness. This is
+therefore a usable **server-side cold-start signal**, and the one metric here that does not
+depend on clients reporting anything.
+
+What it does not include is anything the sandbox does after it is ready — execd bootstrapping
+its own workload, for instance.
+
 ---
 
 ## Environment variables (outside TOML)

server/opensandbox_server/api/lifecycle.py

@@ -48,6 +48,7 @@
 from opensandbox_server.services.constants import SandboxErrorCodes
 from opensandbox_server.services.factory import create_sandbox_service
 from opensandbox_server.services.snapshot_service import create_snapshot_service
+from opensandbox_server.integrations.otel import instrumented_operation
 
 # Initialize router
 router = APIRouter(tags=["Sandboxes"])
@@ -74,6 +75,7 @@
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("create")
 async def create_sandbox(
     request: CreateSandboxRequest,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -214,6 +216,7 @@ def get_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("update_metadata")
 def patch_sandbox_metadata(
     sandbox_id: str,
     patch: PatchSandboxMetadataRequest = Body(...),
@@ -239,6 +242,7 @@ def patch_sandbox_metadata(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("delete")
 def delete_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -279,6 +283,7 @@ def delete_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("pause")
 def pause_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -316,6 +321,7 @@ def pause_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("resume")
 def resume_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -355,6 +361,7 @@ def resume_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("renew")
 def renew_sandbox_expiration(
     sandbox_id: str,
     request: RenewSandboxExpirationRequest,
@@ -402,6 +409,7 @@ def renew_sandbox_expiration(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("create_snapshot")
 def create_snapshot(
     sandbox_id: str,
     response: Response,
@@ -485,6 +493,7 @@ def get_snapshot(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@instrumented_operation("delete_snapshot")
 def delete_snapshot(
     snapshot_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),

server/opensandbox_server/integrations/otel/__init__.py

@@ -12,16 +12,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Optional OpenTelemetry integration for SDK metrics ingestion."""
+"""Optional OpenTelemetry integration for SDK metrics ingestion and server metrics."""
 
+from opensandbox_server.integrations.otel.instrument import instrumented_operation
 from opensandbox_server.integrations.otel.metrics import (
     record_sandbox_create_duration,
+    record_sandbox_operation,
     setup_otel_metrics,
     shutdown_otel_metrics,
 )
 
 __all__ = [
+    "instrumented_operation",
     "record_sandbox_create_duration",
+    "record_sandbox_operation",
     "setup_otel_metrics",
     "shutdown_otel_metrics",
 ]

server/opensandbox_server/integrations/otel/instrument.py

@@ -0,0 +1,110 @@
+# Copyright 2026 Alibaba Group Holding Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Server-side instrumentation for lifecycle API handlers."""
+
+from __future__ import annotations
+
+import functools
+import inspect
+import logging
+import time
+from typing import Any, Callable, Optional
+
+from fastapi import HTTPException
+
+from opensandbox_server.integrations.otel.metrics import record_sandbox_operation
+from opensandbox_server.services.constants import SandboxErrorCodes
+
+logger = logging.getLogger(__name__)
+
+OUTCOME_SUCCESS = "success"
+OUTCOME_ERROR = "error"
+
+
+def _error_code_from(exc: BaseException) -> str:
+    """Pull the server error code out of a raised exception.
+
+    Handlers raise ``HTTPException`` with ``detail={"code": ..., "message": ...}``. When the
+    detail is a plain string, or the exception is not an ``HTTPException`` at all, fall back
+    to the generic code so a failure is still counted rather than dropped.
+    """
+    if isinstance(exc, HTTPException):
+        detail = exc.detail
+        if isinstance(detail, dict):
+            code = detail.get("code")
+            if code is not None:
+                return str(code)
+        return f"HTTP_{exc.status_code}"
+    return SandboxErrorCodes.UNKNOWN_ERROR
+
+
+def instrumented_operation(operation: str) -> Callable[[Callable], Callable]:
+    """Record duration and outcome of a lifecycle handler.
+
+    Measures the server's own work at the API boundary, which is the one place where every
+    runtime converges and where the error code has already been decided. Despite the 202
+    status, ``POST /sandboxes`` blocks until the sandbox is provisioned — Kubernetes awaits
+    ``_wait_for_sandbox_ready``, Docker awaits the provisioning thread — so ``create`` is a
+    cold-start measurement rather than a scheduling one.
+
+    ``functools.wraps`` is load-bearing: FastAPI builds the request model from the handler
+    signature, and ``inspect.signature`` follows ``__wrapped__``, so the route keeps its
+    parameters. Never swallows or alters an exception, and never lets a metrics failure
+    affect the response.
+    """
+
+    def decorator(func: Callable) -> Callable:
+        def _record(started: float, outcome: str, error_code: Optional[str]) -> None:
+            # record_sandbox_operation already swallows instrument errors, but this sits on
+            # every mutating route: a bug in telemetry must not be able to fail a request.
+            try:
+                record_sandbox_operation(
+                    operation=operation,
+                    duration_ms=(time.perf_counter() - started) * 1000.0,
+                    outcome=outcome,
+                    error_code=error_code,
+                )
+            except Exception:
+                logger.exception("Failed to record %s operation metric", operation)
+
+        if inspect.iscoroutinefunction(func):
+
+            @functools.wraps(func)
+            async def async_wrapper(*args: Any, **kwargs: Any) -> Any:
+                started = time.perf_counter()
+                try:
+                    result = await func(*args, **kwargs)
+                except BaseException as exc:
+                    _record(started, OUTCOME_ERROR, _error_code_from(exc))
+                    raise
+                _record(started, OUTCOME_SUCCESS, None)
+                return result
+
+            return async_wrapper
+
+        @functools.wraps(func)
+        def wrapper(*args: Any, **kwargs: Any) -> Any:
+            started = time.perf_counter()
+            try:
+                result = func(*args, **kwargs)
+            except BaseException as exc:
+                _record(started, OUTCOME_ERROR, _error_code_from(exc))
+                raise
+            _record(started, OUTCOME_SUCCESS, None)
+            return result
+
+        return wrapper
+
+    return decorator