feat(server): measure lifecycle operations server-side

docs/guides/sdk-telemetry.md

@@ -45,7 +45,39 @@ After create succeeds or fails, the SDK fire-and-forget posts to `POST /v1/metri
 - `sandboxId` / `image` may be omitted when create fails early.
 - SDK language and version come from the HTTP `User-Agent` header (for example `OpenSandbox-Python-SDK/0.1.15`), not from body fields.
 
-The server accepts the event with `204` and, when `[otel]` is enabled, records an OTEL histogram. See [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
+The server accepts the event with `204` and, when `[otel]` is enabled, records an OTEL histogram (`opensandbox.sandbox.create.duration`). See [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
+
+## SDK-reported vs server-measured
+
+This page describes what the **SDK** reports. `[otel]` also exports metrics the **server**
+measures itself, and the difference matters when you build dashboards:
+
+| Metric | Measured by | Covers |
+|---|---|---|
+| `opensandbox.sandbox.create.duration` | the SDK | the client's whole experience, network and SDK overhead included |
+| `opensandbox.sandbox.operation.duration` | the server | the server's own work at the API boundary |
+| `opensandbox.sandbox.operation.total` | the server | the same, as a count, with the error code on failure |
+
+Two consequences:
+
+- The SDK histogram **only exists where clients report it**. An integration calling the REST
+  API directly produces none of it, so a dashboard built on it alone will look empty rather
+  than healthy. The `operation.*` pair exists in every deployment.
+- `operation.total` carries `error.code` from the server's own taxonomy, which is what makes
+  failures queryable:
+  `sum by (error_code) (rate(opensandbox_sandbox_operation_total{outcome="error"}[5m]))`.
+
+The `operation.*` pair is recorded at the route, so it also counts requests rejected before
+the handler runs — a malformed body appears with `error.code = "HTTP_422"` instead of being
+missing from your error rate.
+
+Both cover provisioning: despite the `202 Accepted` status, `POST /sandboxes` blocks until
+the sandbox exists — Kubernetes awaits pod readiness, Docker awaits container start and
+egress sidecar readiness. So `operation.duration{operation="create"}` is a **server-side
+cold-start signal**, and unlike the SDK histogram it does not depend on clients reporting
+anything. The difference between the two is the client's share: network, auth, SDK overhead.
+
+Full attribute reference: [server configuration](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md#otel).
 
 ## When it runs
 

server/configuration.md

@@ -300,7 +300,7 @@ Per-sandbox enablement uses create request extensions (see OSEP-0009 and `exampl
 
 ## `[otel]`
 
-Optional OpenTelemetry metrics export for SDK-reported sandbox creation latency (`POST /v1/metrics/events`). Off by default; the ingestion endpoint still accepts events and records them as noop.
+Optional OpenTelemetry metrics export. Off by default; the ingestion endpoint still accepts events and records them as noop.
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
@@ -309,6 +309,44 @@ Optional OpenTelemetry metrics export for SDK-reported sandbox creation latency
 | `service_name` | string | `"opensandbox-server"` | `service.name` resource attribute. |
 | `export_interval_millis` | integer | `60000` | Periodic export interval (≥ 1000). |
 
+### Exported metrics
+
+| Metric | Type | Unit | Source |
+|---|---|---|---|
+| `opensandbox.sandbox.create.duration` | Histogram | `ms` | **Reported by the SDK** via `POST /v1/metrics/events` |
+| `opensandbox.sandbox.operation.duration` | Histogram | `ms` | Measured by the server at the API boundary |
+| `opensandbox.sandbox.operation.total` | Counter | - | Measured by the server at the API boundary |
+
+The distinction matters. `create.duration` is a number the client measured and sent, so it
+covers the client's whole experience — network and SDK overhead included — and **only exists
+in deployments whose clients report it**. An integration that calls the REST API directly
+produces none of it.
+
+The `operation.*` pair is the server timing its own work, so it exists in every deployment.
+Attributes:
+
+- `operation` — a lifecycle verb from a closed set: `create`, `update_metadata`, `delete`,
+  `pause`, `resume`, `renew`, `create_snapshot`, `delete_snapshot`. Read-only endpoints are
+  not instrumented.
+- Recorded at the **route**, so it also covers requests rejected before the handler runs — a
+  body that fails validation is counted with `error.code = "HTTP_422"` rather than going
+  missing from the error rate.
+- `outcome` — `success` or `error`.
+- `error.code` — on the counter only, and only when the operation failed. Values come from
+  `SandboxErrorCodes` (`KUBERNETES::POD_READY_TIMEOUT`, `DOCKER::SANDBOX_IMAGE_PULL_FAILED`,
+  and so on), so the server's existing error taxonomy becomes queryable:
+  `sum by (error_code) (rate(opensandbox_sandbox_operation_total{outcome="error"}[5m]))`.
+
+`create` covers **provisioning, not just scheduling**, despite the `202 Accepted` status: the
+handler blocks until the sandbox exists. In Kubernetes `create_sandbox` awaits
+`_wait_for_sandbox_ready`, so the sample includes pod readiness; in Docker it awaits the
+provisioning thread, so it includes container start and egress sidecar readiness. This is
+therefore a usable **server-side cold-start signal**, and the one metric here that does not
+depend on clients reporting anything.
+
+What it does not include is anything the sandbox does after it is ready — execd bootstrapping
+its own workload, for instance.
+
 ---
 
 ## Environment variables (outside TOML)

server/opensandbox_server/api/lifecycle.py

@@ -48,9 +48,10 @@
 from opensandbox_server.services.constants import SandboxErrorCodes
 from opensandbox_server.services.factory import create_sandbox_service
 from opensandbox_server.services.snapshot_service import create_snapshot_service
+from opensandbox_server.integrations.otel import InstrumentedRoute, lifecycle_operation
 
 # Initialize router
-router = APIRouter(tags=["Sandboxes"])
+router = APIRouter(tags=["Sandboxes"], route_class=InstrumentedRoute)
 
 # Initialize service based on configuration from config.toml (defaults to docker)
 sandbox_service = create_sandbox_service()
@@ -74,6 +75,7 @@
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("create")
 async def create_sandbox(
     request: CreateSandboxRequest,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -214,6 +216,7 @@ def get_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("update_metadata")
 def patch_sandbox_metadata(
     sandbox_id: str,
     patch: PatchSandboxMetadataRequest = Body(...),
@@ -239,6 +242,7 @@ def patch_sandbox_metadata(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("delete")
 def delete_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -279,6 +283,7 @@ def delete_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("pause")
 def pause_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -316,6 +321,7 @@ def pause_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("resume")
 def resume_sandbox(
     sandbox_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),
@@ -355,6 +361,7 @@ def resume_sandbox(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("renew")
 def renew_sandbox_expiration(
     sandbox_id: str,
     request: RenewSandboxExpirationRequest,
@@ -402,6 +409,7 @@ def renew_sandbox_expiration(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("create_snapshot")
 def create_snapshot(
     sandbox_id: str,
     response: Response,
@@ -485,6 +493,7 @@ def get_snapshot(
         500: {"model": ErrorResponse, "description": "An unexpected server error occurred"},
     },
 )
+@lifecycle_operation("delete_snapshot")
 def delete_snapshot(
     snapshot_id: str,
     x_request_id: Optional[str] = Header(None, alias="X-Request-ID", description="Unique request identifier for tracing"),

server/opensandbox_server/integrations/otel/__init__.py

@@ -12,16 +12,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Optional OpenTelemetry integration for SDK metrics ingestion."""
+"""Optional OpenTelemetry integration for SDK metrics ingestion and server metrics."""
 
+from opensandbox_server.integrations.otel.instrument import (
+    InstrumentedRoute,
+    lifecycle_operation,
+)
 from opensandbox_server.integrations.otel.metrics import (
     record_sandbox_create_duration,
+    record_sandbox_operation,
     setup_otel_metrics,
     shutdown_otel_metrics,
 )
 
 __all__ = [
+    "InstrumentedRoute",
+    "lifecycle_operation",
     "record_sandbox_create_duration",
+    "record_sandbox_operation",
     "setup_otel_metrics",
     "shutdown_otel_metrics",
 ]

server/opensandbox_server/integrations/otel/instrument.py

@@ -0,0 +1,130 @@
+# Copyright 2026 Alibaba Group Holding Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Server-side instrumentation for lifecycle API routes."""
+
+from __future__ import annotations
+
+import logging
+import time
+from typing import Callable, Optional
+
+from fastapi import HTTPException
+from fastapi.exceptions import RequestValidationError
+from fastapi.routing import APIRoute
+from starlette.requests import Request
+from starlette.responses import Response
+
+from opensandbox_server.integrations.otel.metrics import record_sandbox_operation
+from opensandbox_server.services.constants import SandboxErrorCodes
+
+logger = logging.getLogger(__name__)
+
+OUTCOME_SUCCESS = "success"
+OUTCOME_ERROR = "error"
+
+# Set by lifecycle_operation, read by InstrumentedRoute.
+_OPERATION_ATTRIBUTE = "__opensandbox_operation__"
+
+
+def lifecycle_operation(operation: str) -> Callable[[Callable], Callable]:
+    """Mark a route handler as a lifecycle operation, for ``InstrumentedRoute`` to record.
+
+    A marker rather than a wrapper: recording happens in the route class, which also sees
+    requests rejected before the handler runs. Wrapping here as well would double-count, and
+    would put a layer between FastAPI and the handler signature for no benefit.
+    """
+
+    def decorator(func: Callable) -> Callable:
+        setattr(func, _OPERATION_ATTRIBUTE, operation)
+        return func
+
+    return decorator
+
+
+def _error_code_from(exc: BaseException) -> str:
+    """Pull a bounded error code out of a failure.
+
+    Handlers raise ``HTTPException`` with ``detail={"code": ..., "message": ...}``. Request
+    validation fails before any handler runs and carries no server code, so it is reported by
+    status. Anything unexpected gets the generic code, so a failure is still counted.
+    """
+    if isinstance(exc, RequestValidationError):
+        return "HTTP_422"
+    if isinstance(exc, HTTPException):
+        detail = exc.detail
+        if isinstance(detail, dict):
+            code = detail.get("code")
+            if code is not None:
+                return str(code)
+        return f"HTTP_{exc.status_code}"
+    return SandboxErrorCodes.UNKNOWN_ERROR
+
+
+def _record(operation: str, started: float, outcome: str, error_code: Optional[str]) -> None:
+    """Record one sample, swallowing anything that goes wrong.
+
+    This sits on every mutating route, so a bug in telemetry must not be able to fail a
+    request. ``record_sandbox_operation`` already guards its instruments; this guards the rest.
+    """
+    try:
+        record_sandbox_operation(
+            operation=operation,
+            duration_ms=(time.perf_counter() - started) * 1000.0,
+            outcome=outcome,
+            error_code=error_code,
+        )
+    except Exception:
+        logger.exception("Failed to record %s operation metric", operation)
+
+
+class InstrumentedRoute(APIRoute):
+    """Records duration and outcome for routes marked with ``lifecycle_operation``.
+
+    Instrumenting the route rather than the handler covers the whole API boundary: FastAPI
+    raises ``RequestValidationError`` before the endpoint is called, so a decorator on the
+    handler never sees a malformed request and an error-rate dashboard would undercount it.
+
+    Unmarked routes — the read-only endpoints — get the untouched handler, so nothing is paid
+    for them.
+
+    Note ``POST /sandboxes`` answers 202 but blocks until the sandbox is provisioned
+    (Kubernetes awaits pod readiness, Docker awaits the provisioning thread), so ``create``
+    measures cold start rather than scheduling.
+    """
+
+    def get_route_handler(self) -> Callable:
+        handler = super().get_route_handler()
+        operation: Optional[str] = getattr(self.endpoint, _OPERATION_ATTRIBUTE, None)
+        if operation is None:
+            return handler
+
+        async def instrumented_handler(request: Request) -> Response:
+            started = time.perf_counter()
+            try:
+                response = await handler(request)
+            except BaseException as exc:
+                _record(operation, started, OUTCOME_ERROR, _error_code_from(exc))
+                raise
+            # A handler that returns an error status instead of raising still failed.
+            if response.status_code >= 400:
+                _record(operation, started, OUTCOME_ERROR, f"HTTP_{response.status_code}")
+            else:
+                _record(operation, started, OUTCOME_SUCCESS, None)
+            return response
+
+        return instrumented_handler
+
+
+__all__ = ["InstrumentedRoute", "lifecycle_operation"]