Repository navigation
fix(ssm): run direct executions on virtual threads - #4689
Conversation
Each SendCommand target backed by a Floci EC2 container ran on a cached thread pool, so a command with many targets started one thread per target, each held for the whole container exec (up to TimeoutSeconds). Use a fixed pool of max(4, availableProcessors) daemon threads instead; extra targets wait in the executor queue and still complete.
|
pgermosen
left a comment
There was a problem hiding this comment.
@nkootstra thanks, bounding this makes sense, but a CPU-sized pool changes behaviour in a way I don't think we want. The pool is shared by every command and every instance, and each direct execution holds its thread for the whole exec. On a 4-core runner, four commands that run for a while fill it, and a SendCommand to a completely different instance then sits InProgress without starting until one of them finishes, which with the default TimeoutSeconds can be up to an hour. On main it starts straight away. Since the thread spends that time waiting on the exec, running these on virtual threads (the way CodePipelineService does) would keep every target starting immediately while the waiting costs next to nothing.
If some queue does stay, the start check needs to be stricter too. A queued task only skips Cancelled, but the Auto Scaling reconciler marks invocations on a vanished instance Failed. When the task finally runs, the container is gone, executeIfSupported comes back empty, and the invocation goes back to Pending with a message queued for an instance nothing will poll, sometimes after the command has already rolled up as finished. Skipping anything that isn't still active, and only falling back to Pending while it's still InProgress, would close that.
And the new test waits with a fixed sleep(300). If the bound stays, tracking the peak number of executions in flight and asserting it never goes past the pool size would pin it without the sleep.
pgermosen
left a comment
There was a problem hiding this comment.
@nkootstra virtual threads are the right shape, and the latches replacing the fixed sleep are better than the peak-tracking I suggested.
The other half of my state point is still open though, and virtual threads make it more visible. You guarded the result == null branch, but the success branch still calls applyDirectResult unconditionally. CancelCommand marks an invocation Cancelled without interrupting an in-flight exec, and the check at line 559 only catches a cancel that lands before the exec starts. So cancelling a running SendCommand and letting the exec finish rewrites Cancelled as Success, and the window is now as long as TimeoutSeconds. Same one-line guard as the one just above it.
pgermosen
left a comment
There was a problem hiding this comment.
Nice, the cancel path holds now too. I made one small edit to the description: the exec client still keeps a platform thread per stream, so I reworded the line about blocked execs to say it's the executor's thread that's virtual now. The burst-test cleanup can come later. I put it in #4741, along with the 50-ID limit it would run into. Thanks for sticking with this one.
* fix(ssm): bound direct-execution threads for SendCommand Each SendCommand target backed by a Floci EC2 container ran on a cached thread pool, so a command with many targets started one thread per target, each held for the whole container exec (up to TimeoutSeconds). Use a fixed pool of max(4, availableProcessors) daemon threads instead; extra targets wait in the executor queue and still complete. * fix(ssm): run direct executions on virtual threads * fix(ssm): keep a cancelled invocation cancelled when its exec finishes Signed-off-by: Matheus Oliveira <matheus.6148@gmail.com>
* fix(ssm): bound direct-execution threads for SendCommand Each SendCommand target backed by a Floci EC2 container ran on a cached thread pool, so a command with many targets started one thread per target, each held for the whole container exec (up to TimeoutSeconds). Use a fixed pool of max(4, availableProcessors) daemon threads instead; extra targets wait in the executor queue and still complete. * fix(ssm): run direct executions on virtual threads * fix(ssm): keep a cancelled invocation cancelled when its exec finishes Signed-off-by: Matheus Oliveira <matheus.6148@gmail.com>
Summary
SendCommand runs each target that is a Floci EC2 container directly through a container exec, on
directExecutionExecutor. That executor wasExecutors.newCachedThreadPool, which starts a new platform thread whenever none is idle. Each exec holds its thread until the command finishes or hitsTimeoutSeconds(default 3600), so a command with many targets, or many commands in a row, started one platform thread per target with no upper limit.The executor now starts a virtual thread per target, the same approach
CodePipelineServiceuses. Every target still starts immediately, so one long-running command cannot delay another, and the executor thread waiting on a blocked exec is no longer a platform thread. The container client library still holds one platform thread per exec stream while it reads output.When a direct execution finishes, its result is now only applied while the invocation is still
InProgress. Before, an invocation that changed state during the exec was overwritten:CancelCommand(which does not interrupt the exec), it endedSuccesswhen the exec finished;failActiveInvocationsForInstances, it went back toPendingand was queued when the exec reported unsupported.Type of change
fix:)feat:)feat!:orfix!:)AWS Compatibility
No API change. Invocations still move from
InProgressto their final status, and a cancelled or failed invocation keeps that status.Checklist
./mvnw testpasses locally. Focused suites pass:Ssm*, 218 tests. The full suite was not run locally; CI runs it.SsmCommandServiceDirectExecutionTest.everyDirectExecutionTargetStartsAtOnceOnAVirtualThreadsends one command toavailableProcessors * 4 + 16container targets whose execs block. It waits for every target to be running at once, checks that none runs on a platform thread, then releases them and expectsSuccess.invocationFailedDuringDirectExecutionStaysFailedWhenExecutionFallsBackToAgentQueuefails the invocation while its exec is running, then lets the exec report unsupported. Before the fix the invocation endedPendingwith a queued message.invocationCancelledDuringDirectExecutionStaysCancelledWhenExecutionFinishescancels the command while its exec is running, then lets the exec succeed. Before the fix the invocation endedSuccess.