Skip to content

fix(ssm): run direct executions on virtual threads - #4689

Merged
pgermosen merged 3 commits into
floci-io:mainfrom
nkootstra:fix/ssm-bound-direct-execution-threads
Sep 30, 2026
Merged

pgermosen merged 3 commits into
floci-io:mainfrom
nkootstra:fix/ssm-bound-direct-execution-threads

Conversation

@nkootstra

@nkootstra nkootstra commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

SendCommand runs each target that is a Floci EC2 container directly through a container exec, on directExecutionExecutor. That executor was Executors.newCachedThreadPool, which starts a new platform thread whenever none is idle. Each exec holds its thread until the command finishes or hits TimeoutSeconds (default 3600), so a command with many targets, or many commands in a row, started one platform thread per target with no upper limit.

The executor now starts a virtual thread per target, the same approach CodePipelineService uses. Every target still starts immediately, so one long-running command cannot delay another, and the executor thread waiting on a blocked exec is no longer a platform thread. The container client library still holds one platform thread per exec stream while it reads output.

When a direct execution finishes, its result is now only applied while the invocation is still InProgress. Before, an invocation that changed state during the exec was overwritten:

  • cancelled with CancelCommand (which does not interrupt the exec), it ended Success when the exec finished;
  • failed by the Auto Scaling reconciler through failActiveInvocationsForInstances, it went back to Pending and was queued when the exec reported unsupported.

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

No API change. Invocations still move from InProgress to their final status, and a cancelled or failed invocation keeps that status.

Checklist

  • ./mvnw test passes locally. Focused suites pass: Ssm*, 218 tests. The full suite was not run locally; CI runs it.
  • New or updated integration test added:
    • SsmCommandServiceDirectExecutionTest.everyDirectExecutionTargetStartsAtOnceOnAVirtualThread sends one command to availableProcessors * 4 + 16 container targets whose execs block. It waits for every target to be running at once, checks that none runs on a platform thread, then releases them and expects Success.
    • invocationFailedDuringDirectExecutionStaysFailedWhenExecutionFallsBackToAgentQueue fails the invocation while its exec is running, then lets the exec report unsupported. Before the fix the invocation ended Pending with a queued message.
    • invocationCancelledDuringDirectExecutionStaysCancelledWhenExecutionFinishes cancels the command while its exec is running, then lets the exec succeed. Before the fix the invocation ended Success.
  • Commit messages follow Conventional Commits

Each SendCommand target backed by a Floci EC2 container ran on a cached
thread pool, so a command with many targets started one thread per target,
each held for the whole container exec (up to TimeoutSeconds). Use a fixed
pool of max(4, availableProcessors) daemon threads instead; extra targets
wait in the executor queue and still complete.
@nkootstra
nkootstra marked this pull request as ready for review September 29, 2026 20:24
@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Changes how direct command executions are scheduled and synchronized.

The PR appears safe to merge; no new actionable issue was found.

Summary

The PR moves direct SSM container executions to virtual threads and prevents completed executions from overwriting cancelled or failed invocations.

  • Adds tests for concurrent virtual-thread execution and invocation state preservation.

Reviews (3) · Last reviewed commit: "fix(ssm): keep a cancelled invocation ca..."

@hectorvent hectorvent added bug Something isn't working ssm AWS Systems Manager (SSM) labels Sep 29, 2026

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkootstra thanks, bounding this makes sense, but a CPU-sized pool changes behaviour in a way I don't think we want. The pool is shared by every command and every instance, and each direct execution holds its thread for the whole exec. On a 4-core runner, four commands that run for a while fill it, and a SendCommand to a completely different instance then sits InProgress without starting until one of them finishes, which with the default TimeoutSeconds can be up to an hour. On main it starts straight away. Since the thread spends that time waiting on the exec, running these on virtual threads (the way CodePipelineService does) would keep every target starting immediately while the waiting costs next to nothing.

If some queue does stay, the start check needs to be stricter too. A queued task only skips Cancelled, but the Auto Scaling reconciler marks invocations on a vanished instance Failed. When the task finally runs, the container is gone, executeIfSupported comes back empty, and the invocation goes back to Pending with a message queued for an instance nothing will poll, sometimes after the command has already rolled up as finished. Skipping anything that isn't still active, and only falling back to Pending while it's still InProgress, would close that.

And the new test waits with a fixed sleep(300). If the bound stays, tracking the peak number of executions in flight and asserting it never goes past the pool size would pin it without the sleep.

@pgermosen pgermosen added the waiting-author Review posted; waiting on the PR author to respond label Sep 30, 2026
@nkootstra nkootstra changed the title fix(ssm): bound direct-execution threads for SendCommand fix(ssm): run direct executions on virtual threads Sep 30, 2026

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkootstra virtual threads are the right shape, and the latches replacing the fixed sleep are better than the peak-tracking I suggested.

The other half of my state point is still open though, and virtual threads make it more visible. You guarded the result == null branch, but the success branch still calls applyDirectResult unconditionally. CancelCommand marks an invocation Cancelled without interrupting an in-flight exec, and the check at line 559 only catches a cancel that lands before the exec starts. So cancelling a running SendCommand and letting the exec finish rewrites Cancelled as Success, and the window is now as long as TimeoutSeconds. Same one-line guard as the one just above it.

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, the cancel path holds now too. I made one small edit to the description: the exec client still keeps a platform thread per stream, so I reworded the line about blocked execs to say it's the executor's thread that's virtual now. The burst-test cleanup can come later. I put it in #4741, along with the 50-ID limit it would run into. Thanks for sticking with this one.

@pgermosen pgermosen removed the waiting-author Review posted; waiting on the PR author to respond label Sep 30, 2026
@pgermosen
pgermosen merged commit 6df2a6c into floci-io:main Sep 30, 2026
39 checks passed
@nkootstra
nkootstra deleted the fix/ssm-bound-direct-execution-threads branch September 30, 2026 13:53
omatheusmesmo pushed a commit to omatheusmesmo/floci that referenced this pull request Oct 12, 2026
* fix(ssm): bound direct-execution threads for SendCommand

Each SendCommand target backed by a Floci EC2 container ran on a cached
thread pool, so a command with many targets started one thread per target,
each held for the whole container exec (up to TimeoutSeconds). Use a fixed
pool of max(4, availableProcessors) daemon threads instead; extra targets
wait in the executor queue and still complete.

* fix(ssm): run direct executions on virtual threads

* fix(ssm): keep a cancelled invocation cancelled when its exec finishes
Signed-off-by: Matheus Oliveira <matheus.6148@gmail.com>
omatheusmesmo pushed a commit to omatheusmesmo/floci that referenced this pull request Oct 12, 2026
* fix(ssm): bound direct-execution threads for SendCommand

Each SendCommand target backed by a Floci EC2 container ran on a cached
thread pool, so a command with many targets started one thread per target,
each held for the whole container exec (up to TimeoutSeconds). Use a fixed
pool of max(4, availableProcessors) daemon threads instead; extra targets
wait in the executor queue and still complete.

* fix(ssm): run direct executions on virtual threads

* fix(ssm): keep a cancelled invocation cancelled when its exec finishes
Signed-off-by: Matheus Oliveira <matheus.6148@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ssm AWS Systems Manager (SSM)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants