Skip to content

fix: add timeouts to the terraform/tofu downloads so a stuck fetch cannot hang the executor (#3890) - #112

Merged
alfespa17 merged 2 commits into
terrakube-io:masterfrom
klinux:fix/downloader-timeouts
Nov 11, 2026
Merged

alfespa17 merged 2 commits into
terrakube-io:masterfrom
klinux:fix/downloader-timeouts

Conversation

@klinux

@klinux klinux commented Nov 10, 2026 •

Copy link
Copy Markdown

Fixes the hang described in terrakube-io/terrakube#3890.

Problem

downloadReleasesToFile and the binary download in downloadFileOrReturnPathIfAlreadyExists call .block() with no timeout. If the fetch never completes, the calling thread waits forever. For the releases list this happens while holding RELEASES_LOCKS, so every other caller that needs the list hangs too.

In production, with Netty 4.2 and the Paketo default -XX:MaxDirectMemorySize=10M, Netty raised OutOfDirectMemoryError on the event loop. Reactor's throwIfFatal rethrows it there instead of signalling onError. The Mono never terminates, the job stays in running forever, and the executor pod rejects every new job with this pod is already running a job.

Changes

  • Timeouts, configurable through TerraformProperties (prefix io.terrakube.terraform.flags):

    property default what it caps
    downloadResponseTimeout 60s idle time between reads (responseTimeout, set per request)
    releasesDownloadTimeout 2m whole releases list download (block(Duration))
    binaryDownloadTimeout 10m whole binary download (block(Duration))
    • They flow TerraformProperties → TerraformClient builder → TerraformDownloader, like the releases URLs. null falls back to TerraformDownloader.DEFAULT_*, so existing callers get the defaults.
    • Connecting is already capped by Netty's default 30s connect timeout.
    • Terrakube's executor rebuilds TerraformClient per job and copies only the URLs. For the overrides to take effect there, it needs to copy the three new fields too. The defaults apply either way.
  • OutOfMemoryError is caught around the releases fetch, next to Exception, because OutOfDirectMemoryError extends it. The failure then goes through the existing stale-cache fallback, or turns into the IllegalStateException when there is no cached list. The binary download also catches it, so the partial archive is deleted and an IOException is thrown.

  • One HttpClient / WebClient per process instead of one per call. They live in a lazy holder, so they are created on the first download and not when the class loads. spring-boot-starter-webflux is provided, so class loading behaves as before.

Behaviour change: proxyWithSystemProperties() is now read once, on the first download, instead of on every download.

Tests

TerraformDownloaderTest has three new tests that use an endpoint which never answers:

  • hangingFetchWithoutAPreviousListFailsInsteadOfBlockingForever: expects IllegalStateException.
  • previousListIsUsedWhenARefreshHangs: expects the previous list to be returned.
  • hangingResponseIsCutByTheResponseTimeout: only the response timeout is short, so this checks it is applied per request.

The first two run with a 500ms releases timeout. With .block() restored, both fail by timing out after 10s.

mvn verify -Dgpg.skip: all modules pass, including the starter sample (35 tests in terraform-client).

The OutOfDirectMemoryError path itself is not reproduced in a unit test. It needs a constrained JVM; see the reproduction in the issue. The timeout is what unblocks that case, because the error never reaches block().

🤖 Generated with Claude Code

…nnot hang the executor (#3890)

downloadReleasesToFile and the binary download called block() with no
timeout. When the fetch never completes (seen with Netty 4.2 raising
OutOfDirectMemoryError on the event loop, which Reactor rethrows instead
of signalling onError) the calling thread waits forever while holding
RELEASES_LOCKS, so every other caller of the releases list hangs too.

- add a responseTimeout on the HttpClient and use block(Duration) for
  both the releases list (2 min) and the binary (10 min)
- also catch OutOfMemoryError around the releases fetch so it goes
  through the stale-cache fallback / IllegalStateException
- reuse one HttpClient/WebClient per process instead of one per call

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alfespa17

Copy link
Copy Markdown
Member

instead of hardcoding the values can you add the default values in the properties file so we can override those if need it ?

    private static final Duration RESPONSE_TIMEOUT = Duration.ofSeconds(60);
    static Duration releasesDownloadTimeout = Duration.ofMinutes(2);
    static Duration binaryDownloadTimeout = Duration.ofMinutes(10);

it could be added here https://github.com/terrakube-io/terraform-spring-boot/blob/master/terraform-spring-boot-autoconfigure/src/main/java/io/terrakube/terraform/spring/autoconfigure/TerraformProperties.java

@alfespa17 alfespa17 self-assigned this Nov 11, 2026
…ties

Review feedback: instead of hardcoding them, the timeouts are now
properties with defaults, under io.terrakube.terraform.flags:
downloadResponseTimeout (60s), releasesDownloadTimeout (2m) and
binaryDownloadTimeout (10m). They flow TerraformProperties ->
TerraformClient -> TerraformDownloader like the releases URLs; null
falls back to TerraformDownloader.DEFAULT_*.

The response timeout is now set per request, so the shared HttpClient
can serve downloaders with different settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@klinux

klinux commented Nov 11, 2026

Copy link
Copy Markdown
Author

Done in 031a8d9. The timeouts are now properties in TerraformProperties, with defaults:

io.terrakube.terraform.flags.downloadResponseTimeout=60s
io.terrakube.terraform.flags.releasesDownloadTimeout=2m
io.terrakube.terraform.flags.binaryDownloadTimeout=10m

They go through the TerraformClient builder to TerraformDownloader, same as terraformReleasesUrl / tofuReleasesUrl. When a value is null, the downloader uses its default. I also documented them in the README.

One thing for the Terrakube side: the executor rebuilds TerraformClient per job in TerraformExecutorServiceImpl and copies only the URLs. So to override the timeouts there, those builders also need to copy the three new fields. Without that change the defaults still apply, so the hang is fixed either way. I can send that PR after this one is released.

@alfespa17
alfespa17 merged commit 1d7c4e7 into terrakube-io:master Nov 11, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants