Repository navigation
Conversation
The client's executor was newWorkStealingPool(), a ForkJoinPool sized to availableProcessors(), but every task submitted to it blocks: two stream readers and a waitFor() per command. A ForkJoinPool only grows workers for a ManagedBlocker, which neither readLine() nor waitFor() is, so N concurrent commands need 3N blocked threads and can never have more than availableProcessors(). Starvation begins at 3N-1 > P, so on a four-core machine the second concurrent command already has a reader that is never scheduled. That command delivers no output at all, and once it has written more than a pipe buffer with nobody draining it, the child blocks in write() and stops making progress. A cached pool grows to the blocking work and reaps idle threads after 60s. It is unbounded in principle, but the bound here is the consumer's: three threads per live terraform process, which already costs far more than a thread. Threads are daemon, as ForkJoinPool's workers already were, so JVM exit behaviour is unchanged. everyConcurrentCommandDeliversItsOutput sizes itself to the machine, launching availableProcessors()/3 + 2 commands so that 3N-1 > P holds at any core count without a JVM flag or a surefire change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test class holds its own static work-stealing pool and constructs ProcessLauncher against it directly, so the client's fix does not reach it. It has the same problem for the same reason, and it is why errorStreamIsFullyCapturedWhenExitCodeFutureCompletes cannot pass at one core: that test writes 50000 lines to stderr only, and at parallelism 1 the stdout reader takes the single worker while the stderr reader never runs, so the child blocks on write. With this the suite passes at ActiveProcessorCount 1, 2 and 4. Without it, one core still fails on that test, as it does today. Three lines rather than one because Executors.defaultThreadFactory() creates non-daemon threads while ForkJoinPool's workers are daemon; keeping them daemon preserves what this pool already had. Nothing else in the class depends on work-stealing - timeoutFiresEvenWhenTheLauncherPoolHasNoSpareWorker builds its own ForkJoinPool(1) and never touches the shared pool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
|
@putz612 the build is failing for this one Cc: @jakegroves |
|
Thanks for this, it's a real bug and the self-sizing regression test is great. I reproduced it: your test fails on master and passes on your branch at 1, 2 and the default CPU count. It also triggers with just one command on a 1-CPU container that writes more than 64 KB to stderr (e.g. The branch needed a rebase onto master, and since the fork belongs to an organization I can't push to it. So I opened #114 with your two commits rebased, with your authorship kept. The only conflict was at the end of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pool problem from #101.
TerraformClient's executor is aForkJoinPoolsized toavailableProcessors(), but every task on it is blocking I/O — two readers plus awaitFor()per command — and an FJP only grows workers for aManagedBlocker, which neither is. Starvation starts at3N − 1 > P, so the second concurrent command on a 4-core machine already has a reader that never runs, and its child then blocks inwrite()once it fills the pipe buffer.One field, cached pool, daemon threads as the FJP's workers already were. It's unbounded in principle, but the bound here is the consumer's: three threads per live terraform process, which already costs far more than a thread. Peak threads measured at exactly
3N, reaped by the 60 s keepalive.everyConcurrentCommandDeliversItsOutputsizes itself —availableProcessors() / 3 + 2commands keeps3N − 1 > Pat any core count, so it needs no JVM flag or surefire change. On 4 cores against the current pool:Zero, never a partial count — a reader either gets a thread or it doesn't.
Two behaviour notes:
getExecutor()is public via the class@Getter, so a consumer casting its result now seesThreadPoolExecutorrather thanForkJoinPool(declared type unchanged; I left the method alone rather than break semver in a bug fix). And today's behaviour is an accidental throttle — 200 submitted commands meansPprogressing and the rest wedged inwrite(2); after this they all run.Second commit, drop it if you like.
ProcessLauncherTesthas the same pool in its own harness, which is whyerrorStreamIsFullyCapturedWhenExitCodeFutureCompletescan't pass at one core. With both commits the suite passes atActiveProcessorCount1, 2 and 4; with only the first it still fails at one core, exactly as today. Three lines rather than one becausedefaultThreadFactory()is non-daemon whileForkJoinPool's workers are daemon. Suite wall time unchanged.🤖 Generated with Claude Code