Skip to content

fix: run process I/O on a pool that grows to the blocking work - #103

Open
putz612 wants to merge 2 commits into
terrakube-io:masterfrom
cyberglitchlabs:fix/blocking-io-pool
Open

putz612 wants to merge 2 commits into
terrakube-io:masterfrom
cyberglitchlabs:fix/blocking-io-pool

Conversation

@putz612

@putz612 putz612 commented Sep 14, 2026

Copy link
Copy Markdown

The pool problem from #101.

TerraformClient's executor is a ForkJoinPool sized to availableProcessors(), but every task on it is blocking I/O — two readers plus a waitFor() per command — and an FJP only grows workers for a ManagedBlocker, which neither is. Starvation starts at 3N − 1 > P, so the second concurrent command on a 4-core machine already has a reader that never runs, and its child then blocks in write() once it fills the pipe buffer.

One field, cached pool, daemon threads as the FJP's workers already were. It's unbounded in principle, but the bound here is the consumer's: three threads per live terraform process, which already costs far more than a thread. Peak threads measured at exactly 3N, reaped by the 60 s keepalive.

everyConcurrentCommandDeliversItsOutput sizes itself — availableProcessors() / 3 + 2 commands keeps 3N − 1 > P at any core count, so it needs no JVM flag or surefire change. On 4 cores against the current pool:

with 3 commands on 4 processors the delivered line counts were
stdout=[20000, 20000, 0] of 20000 each;
a command showing 0 had no reader scheduled, and its child is blocked in write()

Zero, never a partial count — a reader either gets a thread or it doesn't.

Two behaviour notes: getExecutor() is public via the class @Getter, so a consumer casting its result now sees ThreadPoolExecutor rather than ForkJoinPool (declared type unchanged; I left the method alone rather than break semver in a bug fix). And today's behaviour is an accidental throttle — 200 submitted commands means P progressing and the rest wedged in write(2); after this they all run.

Second commit, drop it if you like. ProcessLauncherTest has the same pool in its own harness, which is why errorStreamIsFullyCapturedWhenExitCodeFutureCompletes can't pass at one core. With both commits the suite passes at ActiveProcessorCount 1, 2 and 4; with only the first it still fails at one core, exactly as today. Three lines rather than one because defaultThreadFactory() is non-daemon while ForkJoinPool's workers are daemon. Suite wall time unchanged.


🤖 Generated with Claude Code

putz612 and others added 2 commits September 14, 2026 16:10
The client's executor was newWorkStealingPool(), a ForkJoinPool sized to
availableProcessors(), but every task submitted to it blocks: two stream
readers and a waitFor() per command. A ForkJoinPool only grows workers
for a ManagedBlocker, which neither readLine() nor waitFor() is, so N
concurrent commands need 3N blocked threads and can never have more than
availableProcessors().

Starvation begins at 3N-1 > P, so on a four-core machine the second
concurrent command already has a reader that is never scheduled. That
command delivers no output at all, and once it has written more than a
pipe buffer with nobody draining it, the child blocks in write() and
stops making progress.

A cached pool grows to the blocking work and reaps idle threads after
60s. It is unbounded in principle, but the bound here is the consumer's:
three threads per live terraform process, which already costs far more
than a thread. Threads are daemon, as ForkJoinPool's workers already
were, so JVM exit behaviour is unchanged.

everyConcurrentCommandDeliversItsOutput sizes itself to the machine,
launching availableProcessors()/3 + 2 commands so that 3N-1 > P holds at
any core count without a JVM flag or a surefire change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test class holds its own static work-stealing pool and constructs
ProcessLauncher against it directly, so the client's fix does not reach
it. It has the same problem for the same reason, and it is why
errorStreamIsFullyCapturedWhenExitCodeFutureCompletes cannot pass at one
core: that test writes 50000 lines to stderr only, and at parallelism 1
the stdout reader takes the single worker while the stderr reader never
runs, so the child blocks on write.

With this the suite passes at ActiveProcessorCount 1, 2 and 4. Without
it, one core still fails on that test, as it does today.

Three lines rather than one because Executors.defaultThreadFactory()
creates non-daemon threads while ForkJoinPool's workers are daemon;
keeping them daemon preserves what this pool already had. Nothing else
in the class depends on work-stealing -
timeoutFiresEvenWhenTheLauncherPoolHasNoSpareWorker builds its own
ForkJoinPool(1) and never touches the shared pool.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alfespa17

Copy link
Copy Markdown
Member

@putz612 the build is failing for this one

Cc: @jakegroves

@Macbet

Macbet commented Nov 18, 2026

Copy link
Copy Markdown

Thanks for this, it's a real bug and the self-sizing regression test is great. I reproduced it: your test fails on master and passes on your branch at 1, 2 and the default CPU count. It also triggers with just one command on a 1-CPU container that writes more than 64 KB to stderr (e.g. TF_LOG), which is probably the most common way to hit it in practice.

The branch needed a rebase onto master, and since the fork belongs to an organization I can't push to it. So I opened #114 with your two commits rebased, with your authorship kept. The only conflict was at the end of TerraformClientTest, and I kept both sides. I'll close this PR once #114 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants