Skip to content

[codex] stabilize pool scaling - #1425

Open
GodBlf wants to merge 1 commit into
opensandbox-group:mainfrom
GodBlf:fix/1423-pool-scaling-stability
Open

[codex] stabilize pool scaling#1425
GodBlf wants to merge 1 commit into
opensandbox-group:mainfrom
GodBlf:fix/1423-pool-scaling-stability

Conversation

@GodBlf

@GodBlf GodBlf commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • synchronize the Helm Pool CRD with PoolStatus.Updated
  • make pool buffer calculations use Ready, unallocated pods while retaining Pending pods as in-flight capacity
  • bound scale-down batches with scaleStrategy.maxUnavailable and wait for deletions to finish
  • delete least-useful idle pods first and preserve stable Ready capacity
  • count terminating pods against PoolMax
  • keep status updates running while scale expectations are pending

Root cause

Six behaviors amplified one another under slow-starting runtimes: the Helm CRD pruned status.updated, buffer calculations treated Pending pods as available, the buffer target jumped to its midpoint outside the configured band, scale-down was unbounded, terminating pods disappeared from occupancy accounting, and scale-down selected the oldest Ready pods first. Together these caused a hot reconcile loop and sustained pod create/delete oscillation.

Impact

The controller now converges without repeatedly replacing usable pool capacity. Existing Pool manifests remain compatible: no CRD fields, CLI flags, or Helm values were added. The existing scaleStrategy.maxUnavailable setting now limits both scale-up and scale-down batch sizes; the default remains 25%.

Validation

  • KUBEBUILDER_ASSETS="..." go test ./internal/controller -count=1 (36 envtest specs)
  • targeted pool scaling stability unit tests
  • go test ./internal/utils/expectations/... -count=1
  • go test ./... -run '^$'
  • go vet ./internal/controller ./internal/utils/expectations/...
  • helm lint charts/opensandbox-controller
  • rendered Helm CRD contains .status.updated and the UPDATED printer column
  • pnpm docs:build
  • git diff --check

Closes #1423

Copilot AI review requested due to automatic review settings July 30, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added component/k8s For kubernetes runtime documentation Improvements or additions to documentation size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 30, 2026
@GodBlf
GodBlf marked this pull request as ready for review July 30, 2026 09:40

@Pangjiping Pangjiping left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @fengcone @Spground ptal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/k8s For kubernetes runtime documentation Improvements or additions to documentation size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pool controller: hot reconcile loop + unthrottled scale-down oscillation creates/deletes ~2200 pods per minute, AVAILABLE stuck at 0

6 participants