feat(kotlin): trace sandbox pool warmup with OpenTelemetry

docs/.vitepress/config.mts

@@ -99,6 +99,7 @@ export default defineConfig({
             { text: "Windows Sandbox", link: "/guides/windows-sandbox" },
             { text: "Client Pool", link: "/guides/client-pool" },
             { text: "SDK Telemetry", link: "/guides/sdk-telemetry" },
+            { text: "SDK Tracing (Pool Warmup)", link: "/guides/sdk-tracing" },
           ],
         },
       ],

docs/guides/client-pool.md

@@ -296,6 +296,15 @@ validate a positive concurrency value and wait for every drained ID to receive a
 best-effort kill attempt. The Go method is intentionally outside the
 `SandboxPool` interface to preserve compatibility with third-party implementors.
 
+### Tracing warmups (Kotlin)
+
+The Kotlin SDK can emit an OpenTelemetry trace per warmup task (`pool.warmup`
+root span plus `create` / `prepare` / `renew` / `commit` phases) when
+`ConnectionConfig.enableTracing(true)` is set and an OpenTelemetry SDK +
+exporter is on the classpath. `trace_id` / `span_id` are published to the
+SLF4J MDC, so search your logs for a `sandbox_id` to find the warmup trace and
+drill into phase durations. See [SDK Tracing (Pool Warmup)](/guides/sdk-tracing).
+
 ### Retiring an old pool namespace
 
 The retirement procedure differs across SDKs because Go does not currently ship a

docs/guides/sdk-tracing.md

@@ -0,0 +1,182 @@
+---
+title: SDK Tracing (Pool Warmup)
+description: How to enable OpenTelemetry tracing for the Kotlin SDK pool warmup path, what spans are produced, and how to query and drill down into warmup traces.
+---
+
+# SDK Tracing (Pool Warmup)
+
+The Kotlin/Java SDK (`com.alibaba.opensandbox:sandbox`) can emit
+[OpenTelemetry](https://opentelemetry.io/) traces for the client-side
+`SandboxPool` warmup path. Each warmup task becomes one trace that covers the
+full lifecycle — from the moment the reconcile loop submits the task until the
+warmed sandbox is committed to the idle buffer — with per-phase spans so you
+can find the actual warmup bottleneck.
+
+Tracing is **opt-in** (`enableTracing(true)`) and **best-effort**: without an
+OpenTelemetry SDK + exporter on the application classpath, all span calls are
+no-ops and nothing is exported. Tracing never throws and never affects pool
+behavior.
+
+## Requirements
+
+| Component | Minimum version |
+|-----------|-----------------|
+| Kotlin / Java SDK (`com.alibaba.opensandbox:sandbox`) | `1.0.19` |
+
+Only the Kotlin SDK emits these traces today; the other language SDKs do not
+yet support `enableTracing`.
+
+## Enabling tracing
+
+### 1. Add an OpenTelemetry SDK + exporter to your application
+
+The SDK depends only on `opentelemetry-api` (no-op by default). To actually
+export traces you bring your own SDK and exporter, for example OTLP over HTTP:
+
+```kotlin
+dependencies {
+    implementation("io.opentelemetry:opentelemetry-api:1.51.0")
+    implementation("io.opentelemetry:opentelemetry-sdk:1.51.0")
+    implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.51.0")
+}
+```
+
+### 2. Configure a global `OpenTelemetry` instance
+
+Warmup spans use the global instance (`GlobalOpenTelemetry`). Configure it at
+application startup, e.g. with `OpenTelemetrySdk`:
+
+```java
+import io.opentelemetry.sdk.OpenTelemetrySdk;
+import io.opentelemetry.sdk.trace.SdkTracerProvider;
+import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
+import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
+
+SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
+    .addSpanProcessor(BatchSpanProcessor.create(
+        OtlpGrpcSpanExporter.builder()
+            .setEndpoint("http://otel-collector:4317")
+            .build()))
+    .build();
+
+OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
+    .setTracerProvider(tracerProvider)
+    .build();
+
+GlobalOpenTelemetry.set(sdk);
+```
+
+::: tip Propagators
+`OpenTelemetrySdk.builder()` defaults to **noop propagators**. If you want the
+SDK to inject the W3C `traceparent` header into lifecycle requests (so the
+lifecycle server can join the same trace once it supports tracing), configure
+W3C propagation explicitly:
+
+```java
+.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
+```
+:::
+
+::: tip Sampling
+To keep trace volume bounded, use a sampling strategy such as
+`parentbased_traceidratio(0.1)` on the `SdkTracerProvider`. Trace-id-ratio
+sampling keeps client and server spans consistent for the same warmup.
+:::
+
+### 3. Turn tracing on for the pool
+
+```java
+ConnectionConfig config = ConnectionConfig.builder()
+    .enableTracing(true)
+    .build();
+
+SandboxPool pool = SandboxPool.builder()
+    .poolName("demo-pool")
+    .maxIdle(3)
+    .stateStore(new InMemoryPoolStateStore())
+    .connectionConfig(config)
+    .creationSpec(PoolCreationSpec.builder().image("ubuntu:22.04").build())
+    .build();
+```
+
+That is all. No environment variables are involved; `enableTracing` defaults
+to `false`.
+
+## What is traced
+
+Each warmup task produces **one trace** with a root span and four sequential
+phase spans (siblings under the root, so each phase duration stands alone for
+comparison):
+
+| Span name | Covers |
+|-----------|--------|
+| `pool.warmup` (root) | Task submission → sandbox committed to idle. Backdated to submission time, so the queue wait before the first phase is visible as the gap before the first child span |
+| `pool.warmup.create` | Sandbox create API call, endpoint resolution, and readiness wait |
+| `pool.warmup.prepare` | The configured `warmupSandboxPreparer` (user init script / setup work) |
+| `pool.warmup.renew` | TTL renewal right before committing the sandbox |
+| `pool.warmup.commit` | Primary-lock renewal + `putIdle` against the state store (runs on the pool scheduler thread) |
+
+Root span attributes (these are your drill-down dimensions):
+
+| Attribute | Value |
+|-----------|-------|
+| `pool.name` | Pool name |
+| `pool.owner` | Pool owner id |
+| `pool.run.generation` | Pool run generation |
+| `sandbox.id` | Sandbox id (success only) |
+| `sandbox.image` | Creation image (success only) |
+| `result` | `success` or `failure` |
+
+Failures are recorded with `recordException` on the root span plus
+`result=failure`; the `pool.warmup.commit` span is not emitted for failed
+warmups.
+
+## Correlating logs to traces
+
+While a warmup trace is in progress, the pool publishes the trace ids to the
+SLF4J [MDC](https://www.slf4j.org/api/org/slf4j/MDC.html):
+
+| MDC key | Value |
+|---------|-------|
+| `trace_id` | Current trace id |
+| `span_id` | Current span id |
+
+MDC requires a real SLF4J provider (logback, log4j2, ...). Add the keys to
+your log pattern once, and every pool log line carries the trace context:
+
+```xml
+<pattern>%d %-5level [%thread] %logger{36} trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n</pattern>
+```
+
+## Querying traces
+
+The trace id is random, so a warmup trace cannot be looked up "by pool name"
+directly. The reliable paths are:
+
+1. **Log correlation (recommended).** The pool already logs `pool_name` and
+   `sandbox_id` on its warmup lines (e.g. `Pool warmup sandbox entered idle`).
+   Search your logs for a `sandbox_id` — the matching log lines carry
+   `trace_id`, which you can open directly in your trace backend.
+2. **Attribute query in the trace backend.** Filter spans by time window and
+   attribute, e.g. TraceQL `{ span.pool.name = "demo-pool" }` (Grafana Tempo),
+   or Jaeger tag search on `pool.name=...`. Backends that derive metrics from
+   spans (Tempo metrics, Datadog span analytics) let you look at
+   `pool.warmup` duration percentiles per `pool.name` first, then drill into
+   slow traces.
+3. **Trace-id-ratio sampling.** With sampled traces, `trace_id` in logs and
+   the backend are consistent for the same warmup.
+
+### Bottleneck drill-down
+
+```
+pool.warmup root duration (p50/p95/p99) per pool.name
+  └─ phase spans: pool.warmup.create / prepare / renew / commit
+       └─ single trace: root start gap = queue wait, then each phase duration
+```
+
+| Symptom | Likely cause |
+|---------|--------------|
+| Long gap before the first child span | Warmup tasks queued — `warmupConcurrency` too low, or the executor is busy with cleanup kills |
+| `pool.warmup.create` slow | Lifecycle server slow (image pull / execd startup) or readiness polling takes long |
+| `pool.warmup.prepare` slow | Your `warmupSandboxPreparer` work is the bottleneck |
+| `pool.warmup.renew` / `pool.warmup.commit` slow | State store (e.g. Redis) round-trips |

docs/sdks/kotlin.md

@@ -301,6 +301,14 @@ poolManager.destroy(
 - Use `warmupSandboxPreparer(...)` if you need to prepare a sandbox after warmup readiness succeeds and before it is put into the idle pool.
 :::
 
+::: tip Observing warmup performance
+To trace the warmup path, enable `ConnectionConfig.builder().enableTracing(true)` and add an
+OpenTelemetry SDK + exporter to your application. Each warmup becomes one trace
+(`pool.warmup` root span plus `create` / `prepare` / `renew` / `commit` phases) with
+`trace_id` / `span_id` published to the SLF4J MDC, so you can look up a sandbox's
+warmup by searching logs for its `sandbox_id`. See [SDK Tracing (Pool Warmup)](/guides/sdk-tracing).
+:::
+
 ::: tip Distributed Deployment
 For distributed deployment, use the optional `com.alibaba.opensandbox:sandbox-pool-redis` module or provide a custom `PoolStateStore` implementation. The Redis module accepts a caller-managed Jedis client, so your application keeps ownership of Redis connection configuration and lifecycle. Nodes sharing the same pool namespace must use the same sandbox creation and warmup definition; use a new `poolName` or namespace when changing that definition. Configure `primaryLockTtl` greater than `warmupReadyTimeout` plus expected warmup preparer time and buffer, otherwise leadership may expire while a node is creating idle sandboxes.
 
@@ -329,6 +337,7 @@ The `ConnectionConfig` class manages API server connection settings.
 | `retryPolicy`    | Automatic retry policy for non-streaming requests (see [Automatic retries](#_2-automatic-retries)) | Enabled (`RetryPolicy()`) | -                 |
 | `useServerProxy` | Use sandbox server as proxy for execd/endpoint requests (e.g. when client cannot reach the sandbox directly) | `false` | -                      |
 | `disableMetrics` | Disable SDK create-latency telemetry (see [SDK Telemetry](/guides/sdk-telemetry)) | `false` | `OPENSANDBOX_DISABLE_METRICS` |
+| `enableTracing` | Enable OpenTelemetry tracing for pool warmup (see [SDK Tracing](/guides/sdk-tracing)) | `false` | - |
 
 ```java
 // 1. Basic configuration

sdks/sandbox/kotlin/gradle/libs.versions.toml

@@ -17,6 +17,7 @@ kotlin = "2.2.21"
 kotlinx-serialization = "1.9.0"
 okhttp = "4.12.0"
 slf4j = "2.0.9"
+opentelemetry = "1.51.0"
 jedis = "5.2.0"
 junit = "5.10.1"
 mockk = "1.13.8"
@@ -43,6 +44,11 @@ kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serializa
 
 # Logging
 slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }
+logback-classic = { module = "ch.qos.logback:logback-classic", version = "1.5.18" }
+
+# Tracing (API only; no-op when no OpenTelemetry SDK is on the classpath)
+opentelemetry-api = { module = "io.opentelemetry:opentelemetry-api", version.ref = "opentelemetry" }
+opentelemetry-sdk-testing = { module = "io.opentelemetry:opentelemetry-sdk-testing", version.ref = "opentelemetry" }
 
 # Redis
 jedis = { module = "redis.clients:jedis", version.ref = "jedis" }

sdks/sandbox/kotlin/sandbox-bom/build.gradle.kts

@@ -29,5 +29,6 @@ dependencies {
         api(libs.okhttp)
         api(libs.okhttp.logging)
         api(libs.slf4j.api)
+        api(libs.opentelemetry.api)
     }
 }

sdks/sandbox/kotlin/sandbox/build.gradle.kts

@@ -21,10 +21,13 @@ dependencies {
 
     implementation(libs.okhttp)
     implementation(libs.okhttp.logging)
+    implementation(libs.opentelemetry.api)
     compileOnly(libs.bundles.serialization)
 
     testImplementation(libs.bundles.testing)
     testImplementation(libs.bundles.serialization)
+    testImplementation(libs.opentelemetry.sdk.testing)
+    testRuntimeOnly(libs.logback.classic)
     testRuntimeOnly(libs.junit.platform.launcher)
 }
 

sdks/sandbox/kotlin/sandbox/src/main/kotlin/com/alibaba/opensandbox/sandbox/HttpClientProvider.kt

@@ -19,7 +19,12 @@ package com.alibaba.opensandbox.sandbox
 import com.alibaba.opensandbox.sandbox.config.ConnectionConfig
 import com.alibaba.opensandbox.sandbox.domain.models.execd.SECURE_ACCESS_HEADER
 import com.alibaba.opensandbox.sandbox.transport.RetryInterceptor
+import io.opentelemetry.api.GlobalOpenTelemetry
+import io.opentelemetry.context.Context
+import io.opentelemetry.context.propagation.TextMapPropagator
+import io.opentelemetry.context.propagation.TextMapSetter
 import okhttp3.ConnectionPool
+import okhttp3.Headers
 import okhttp3.Interceptor
 import okhttp3.OkHttpClient
 import okhttp3.Response
@@ -44,12 +49,21 @@ class HttpClientProvider(
     private val connectionPoolOwnedBySdk: Boolean = config.connectionPool == null
 
     private val baseBuilder: OkHttpClient.Builder
-        get() =
-            OkHttpClient.Builder()
-                .connectionPool(connectionPool)
-                .addInterceptor(UserAgentInterceptor(config.userAgent))
-                .addInterceptor(ExtraHeadersInterceptor(config.headers))
-                .addInterceptor(ClientIpInterceptor { ClientIpDetector.clientIp() })
+        get() {
+            val builder =
+                OkHttpClient.Builder()
+                    .connectionPool(connectionPool)
+                    .addInterceptor(UserAgentInterceptor(config.userAgent))
+                    .addInterceptor(ExtraHeadersInterceptor(config.headers))
+                    .addInterceptor(ClientIpInterceptor { ClientIpDetector.clientIp() })
+            if (config.enableTracing) {
+                // Propagate the active trace context (W3C traceparent) so the
+                // lifecycle server can join the same trace. No-op when there
+                // is no active span in the current context.
+                builder.addInterceptor(TraceContextInterceptor(GlobalOpenTelemetry.getPropagators().textMapPropagator))
+            }
+            return builder
+        }
 
     // 1. Explicit lazy definition to allow checking initialization status
     private val httpClientLazy =
@@ -202,6 +216,27 @@ class HttpClientProvider(
         }
     }
 
+    /**
+     * Injects the W3C `traceparent` / `tracestate` headers of the current
+     * OpenTelemetry context into every request. When no span is active the
+     * propagator injects nothing and the request passes through unchanged.
+     */
+    private class TraceContextInterceptor(
+        private val propagators: TextMapPropagator,
+    ) : Interceptor {
+        private val setter =
+            TextMapSetter<Headers.Builder> { carrier: Headers.Builder?, key, value ->
+                carrier?.set(key, value)
+            }
+
+        override fun intercept(chain: Interceptor.Chain): Response {
+            val request = chain.request()
+            val headers = request.headers.newBuilder()
+            propagators.inject(Context.current(), headers, setter)
+            return chain.proceed(request.newBuilder().headers(headers.build()).build())
+        }
+    }
+
     // --- Cleanup ---
 
     /**

sdks/sandbox/kotlin/sandbox/src/main/kotlin/com/alibaba/opensandbox/sandbox/config/ConnectionConfig.kt

@@ -59,6 +59,17 @@ class ConnectionConfig private constructor(
      * Also honored via the `OPENSANDBOX_DISABLE_METRICS=1` environment variable.
      */
     val disableMetrics: Boolean = false,
+    /**
+     * Enable OpenTelemetry tracing for the client-side sandbox pool warmup path.
+     *
+     * Off by default. When enabled, each pool warmup creates an OpenTelemetry
+     * trace (`pool.warmup` root span plus per-phase spans) and the active
+     * trace context is propagated to lifecycle requests via the W3C
+     * `traceparent` header. Tracing is best-effort: without an
+     * OpenTelemetry SDK + exporter on the classpath, all span calls are
+     * no-ops and nothing is exported.
+     */
+    val enableTracing: Boolean = false,
     /**
      * Retry policy applied to non-streaming requests. Enabled by default; pass
      * [RetryPolicy.disabled] to disable SDK-policy retries and fall back to
@@ -88,6 +99,7 @@ class ConnectionConfig private constructor(
             endpointCacheSize = this.endpointCacheSize,
             endpointCacheDisabled = this.endpointCacheDisabled,
             disableMetrics = this.disableMetrics,
+            enableTracing = this.enableTracing,
             retryPolicy = this.retryPolicy,
         )
 
@@ -188,6 +200,7 @@ class ConnectionConfig private constructor(
         private var endpointCacheSize: Int = 1024
         private var endpointCacheDisabled: Boolean = false
         private var disableMetrics: Boolean = false
+        private var enableTracing: Boolean = false
         private var retryPolicy: RetryPolicy = RetryPolicy()
 
         /**
@@ -228,6 +241,17 @@ class ConnectionConfig private constructor(
             return this
         }
 
+        /**
+         * Enable OpenTelemetry tracing for the client-side sandbox pool warmup path.
+         *
+         * Off by default; pass `true` to opt in. Tracing is best-effort and no-ops
+         * unless an OpenTelemetry SDK + exporter is on the classpath.
+         */
+        fun enableTracing(enable: Boolean = true): Builder {
+            this.enableTracing = enable
+            return this
+        }
+
         /**
          * Set the API key used for authentication.
          *
@@ -383,6 +407,7 @@ class ConnectionConfig private constructor(
                 endpointCacheSize = endpointCacheSize,
                 endpointCacheDisabled = endpointCacheDisabled,
                 disableMetrics = disableMetrics,
+                enableTracing = enableTracing,
                 retryPolicy = retryPolicy,
             )
         }