fix: cap inference-pod CPU threads and offload blocking event-loop calls by honeytung · Pull Request #442 · groundlight/edge-endpoint

honeytung · 2026-06-29T21:58:32Z

Summary

Root cause: inference pods had no CPU thread limits, so each pod spawned ~112 OMP threads. With N detector copies = 2N pods, total threads exceeded 1000 on a 112-core machine — classic congestion collapse. Throughput peaked at 88 req/s (3 copies) then fell while CPU cost per request grew 5×; GPUs stayed at 2–5%.
Deployment fix: cap the inference pod's CPU threads via a configurable Helm value — inferenceDeployment.cpuThreads (default 4) sets OMP_NUM_THREADS/MKL_NUM_THREADS/OPENBLAS_NUM_THREADS. Inference pods set no CPU requests or limits (BestEffort), so the scheduler launches every detector regardless of core count; threads are bounded by cpuThreads, which also avoids CFS throttling.
App fix: add timeout=2 to the no-timeout is_edge_inference_ready health check; offload get_detector_metadata, inference_is_available, and run_inference from the async event loop via run_in_threadpool; make get_app_state async def to remove the pointless threadpool dispatch on every request.

Changes

deploy/helm/groundlight-edge-endpoint/files/inference-deployment-template.yaml

Set OMP_NUM_THREADS / MKL_NUM_THREADS / OPENBLAS_NUM_THREADS from inferenceDeployment.cpuThreads (default 4) — caps each pod's intra-op thread pool so N pods don't oversubscribe the cores. Tune per device.
Set no CPU requests or limits on the inference pods (BestEffort), so the scheduler launches every detector regardless of core count (detectors are mostly idle; active pods stay bounded by cpuThreads, and the OS time-slices simultaneous bursts). No CFS throttling.

deploy/helm/groundlight-edge-endpoint/values.yaml

Add inferenceDeployment.cpuThreads (default 4) so the thread cap is tunable per device without editing the chart.

app/core/edge_inference.py

Add timeout=2 to requests.get() in is_edge_inference_ready — previously no timeout; a briefly-unreachable pod could block the event loop for up to 60s. Timeout is a subclass of RequestException, so the existing except already handles it.

app/api/routes/image_queries.py

await run_in_threadpool(get_detector_metadata, ...) — was a sync network call on the event loop (cached after first call, but first call can take 2–3s).
await run_in_threadpool(inference_is_available, ...) — was 2 sync requests.get health checks on the event loop, each previously with no timeout (p99=1023ms in profiling).
await run_in_threadpool(run_inference, ...) — was blocking f_primary.result() / f_oodd.result() waits on the event loop.

app/core/app_state.py

get_app_state: def → async def — the function is a pure attribute read; FastAPI was dispatching it to a threadpool on every single request unnecessarily.

Test plan

Run make test — 342 unit tests pass, pre-existing PermissionError fixture failures are unrelated to these changes
On device: kubectl exec -n edge <inference-pod> -- sh -c 'echo OMP=$OMP_NUM_THREADS; python -c "import torch; print(torch.get_num_threads())"' — expect 4
On device: confirm the rendered inference template caps threads (OMP_NUM_THREADS=4) and sets no CPU resources block (BestEffort)
Re-run the lens benchmark sweep (copies: [1,2,3,4,5], target_fps: 0) and confirm throughput no longer collapses past 3 copies, per-request CPU cost stays roughly flat, and GPU% rises

🤖 Generated with Claude Code

Validated on hardware

Device avc2-profiling (2× Xeon Gold 5520+ / 112 threads, 4× RTX Pro 4000 Blackwell). Lens benchmark bbox_to_binary, objects=5, target_fps=0 (saturate), copies 1→5:

metric (at 5 copies / saturating)	pre	post
total throughput	72 req/s	245 req/s (3.4×)
scaling vs detectors	plateaus then collapses (8.4→14.6→12.0 FPS)	scales linearly (9.9→40.8 FPS)
p95 latency	281 ms	74 ms (3.8× lower)
CPU cost / request	1.25 core-sec	0.04 core-sec (32× less)
node CPU	80%	8%
GPU utilization	4.6%	11.3%

Root cause was PyTorch/OpenMP thread oversubscription in the inference pods (no thread cap → each pod spawned ~112 threads; N pods thrashed the cores). The thread cap + event-loop offload eliminate it; per-request CPU cost is now flat with load (the oversubscription signature is gone) and the box is no longer CPU-bound.

…ent loop Root cause: inference pods had no CPU thread limits, causing each pod to spin up ~112 OMP threads. With N detector copies = 2N pods, total thread count exceeded 1000 on a 112-core machine, causing congestion collapse past 3 copies (throughput peaked at ~88 req/s then fell, CPU cost per request grew 5×). Deployment changes (inference-deployment-template.yaml): - Add OMP_NUM_THREADS/MKL_NUM_THREADS/OPENBLAS_NUM_THREADS=4 env vars to cap intra-op thread pools to a sensible per-pod budget (112 cores / ~20-30 pods) - Add resources requests/limits (cpu: 4/8) so the scheduler spreads pods and cgroup-aware PyTorch sizes its pool correctly App changes: - Add timeout=2 to is_edge_inference_ready() requests.get — previously no timeout meant a briefly-unreachable pod could block the event loop for ~60s - Offload get_detector_metadata, inference_is_available, and run_inference from async post_image_query via run_in_threadpool — all three are sync-blocking network/thread-wait calls that were running directly on the event loop - Make get_app_state async def — it does only an attribute read; the sync def was causing an unnecessary threadpool dispatch on every single request Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Make the inference-pod CPU thread cap and CPU request Helm values instead of hardcoded, and remove the CPU limit: - inferenceDeployment.cpuThreads (default 4) -> OMP/MKL/OPENBLAS_NUM_THREADS - inferenceDeployment.cpuRequest (default 1) -> resources.requests.cpu - removed resources.limits.cpu (threads are bounded by cpuThreads; omitting the limit avoids CFS throttling / tail-latency spikes) The previous hardcoded requests.cpu="4" reserved 4 cores per pod, capping a 112-core node at ~13 detectors (28 pods); beyond that, pods stayed Pending and set_config timed out waiting for readiness. requests.cpu="1" raises the ceiling to ~55 detectors while threads stay capped at cpuThreads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Default inferenceDeployment.cpuRequest to "" so the resources.requests block is omitted entirely => inference pods are BestEffort and the scheduler launches all detectors regardless of core count. Rationale: most detectors are idle at any instant, active pods are already bounded by cpuThreads (OMP/MKL/OpenBLAS), and the OS time-slices simultaneous bursts. A non-empty cpuRequest still reserves CPU and caps density for nodes that want a hard scheduling guarantee. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Load testing showed BestEffort (no CPU request) is the right behavior in every case: threads are bounded by cpuThreads, detectors are mostly idle at any instant, and the scheduler should launch all of them regardless of core count. The optional cpuRequest knob is unneeded, so drop it along with the conditional resources block. Inference pods now set no CPU requests or limits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

timmarkhuff · 2026-07-01T23:34:50Z

    # Ensure that detector_id has correct casing by pulling the detector ID out of detector_metadata
    # get_detector_metadata returns the correctly-cased, canonical detector ID
-    detector_metadata = get_detector_metadata(detector_id=detector_id, gl=gl)  # NOTE: API call (once, then cached)
+    detector_metadata = await run_in_threadpool(


This seems unnecessary, but maybe it doesn't hurt.

I ran a load test with this code and didn't see any issues.

timmarkhuff · 2026-07-01T23:36:22Z

        logger.debug("Received human_review=ALWAYS. Skipping edge inference.")
        record_activity_for_metrics(detector_id, activity_type="escalations")
-    elif app_state.edge_inference_manager.inference_is_available(detector_id=detector_id):
+    elif await run_in_threadpool(app_state.edge_inference_manager.inference_is_available, detector_id=detector_id):


Eventually I would like to get rid of this readiness cache. I have a PR that should do the trick: #419

But for now this seem fine.

timmarkhuff · 2026-07-01T23:36:31Z

          value: "{{ .Values.edgeCompileEnabled }}"
+        # Cap PyTorch/OpenMP/MKL/OpenBLAS intra-op threads so that many inference pods
+        # don't oversubscribe the node's cores. See values.yaml inferenceDeployment.cpuThreads.
+        - name: OMP_NUM_THREADS


timmarkhuff

LGTM

honeytung requested a review from a team as a code owner June 29, 2026 21:58

Auto-format Bot and others added 3 commits June 29, 2026 21:58

Automatically reformatting code with black and isort

1183b18

honeytung force-pushed the edge-endpoint-perf-implementation branch from f49bf9c to 25f8e02 Compare June 30, 2026 17:43

Merge branch 'main' into edge-endpoint-perf-implementation

9b72616

honeytung requested review from f-wright and timmarkhuff July 1, 2026 19:58

timmarkhuff reviewed Jul 1, 2026

View reviewed changes

timmarkhuff approved these changes Jul 1, 2026

View reviewed changes

honeytung merged commit d4ae5fe into main Jul 2, 2026
14 checks passed

honeytung deleted the edge-endpoint-perf-implementation branch July 2, 2026 00:05

Navigation Menu

Uh oh!

fix: cap inference-pod CPU threads and offload blocking event-loop calls - #442

fix: cap inference-pod CPU threads and offload blocking event-loop calls#442
honeytung merged 6 commits into
mainfrom
edge-endpoint-perf-implementation

honeytung commented Jun 29, 2026 •

edited

Loading

Uh oh!

timmarkhuff Jul 1, 2026

Uh oh!

timmarkhuff Jul 1, 2026

Uh oh!

timmarkhuff Jul 1, 2026

Uh oh!

timmarkhuff left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

honeytung commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test plan

Validated on hardware

Uh oh!

timmarkhuff Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

timmarkhuff Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

timmarkhuff Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

timmarkhuff left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

honeytung commented Jun 29, 2026 •

edited

Loading