Repository navigation
fix: add timeouts to the terraform/tofu downloads so a stuck fetch cannot hang the executor (#3890) - #112
Conversation
…nnot hang the executor (#3890) downloadReleasesToFile and the binary download called block() with no timeout. When the fetch never completes (seen with Netty 4.2 raising OutOfDirectMemoryError on the event loop, which Reactor rethrows instead of signalling onError) the calling thread waits forever while holding RELEASES_LOCKS, so every other caller of the releases list hangs too. - add a responseTimeout on the HttpClient and use block(Duration) for both the releases list (2 min) and the binary (10 min) - also catch OutOfMemoryError around the releases fetch so it goes through the stale-cache fallback / IllegalStateException - reuse one HttpClient/WebClient per process instead of one per call Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
instead of hardcoding the values can you add the default values in the properties file so we can override those if need it ? it could be added here https://github.com/terrakube-io/terraform-spring-boot/blob/master/terraform-spring-boot-autoconfigure/src/main/java/io/terrakube/terraform/spring/autoconfigure/TerraformProperties.java |
…ties Review feedback: instead of hardcoding them, the timeouts are now properties with defaults, under io.terrakube.terraform.flags: downloadResponseTimeout (60s), releasesDownloadTimeout (2m) and binaryDownloadTimeout (10m). They flow TerraformProperties -> TerraformClient -> TerraformDownloader like the releases URLs; null falls back to TerraformDownloader.DEFAULT_*. The response timeout is now set per request, so the shared HttpClient can serve downloaders with different settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Done in 031a8d9. The timeouts are now properties in They go through the One thing for the Terrakube side: the executor rebuilds |
Fixes the hang described in terrakube-io/terrakube#3890.
Problem
downloadReleasesToFileand the binary download indownloadFileOrReturnPathIfAlreadyExistscall.block()with no timeout. If the fetch never completes, the calling thread waits forever. For the releases list this happens while holdingRELEASES_LOCKS, so every other caller that needs the list hangs too.In production, with Netty 4.2 and the Paketo default
-XX:MaxDirectMemorySize=10M, Netty raisedOutOfDirectMemoryErroron the event loop. Reactor'sthrowIfFatalrethrows it there instead of signallingonError. TheMononever terminates, the job stays inrunningforever, and the executor pod rejects every new job withthis pod is already running a job.Changes
Timeouts, configurable through
TerraformProperties(prefixio.terrakube.terraform.flags):downloadResponseTimeout60sresponseTimeout, set per request)releasesDownloadTimeout2mblock(Duration))binaryDownloadTimeout10mblock(Duration))TerraformProperties→TerraformClientbuilder →TerraformDownloader, like the releases URLs.nullfalls back toTerraformDownloader.DEFAULT_*, so existing callers get the defaults.TerraformClientper job and copies only the URLs. For the overrides to take effect there, it needs to copy the three new fields too. The defaults apply either way.OutOfMemoryErroris caught around the releases fetch, next toException, becauseOutOfDirectMemoryErrorextends it. The failure then goes through the existing stale-cache fallback, or turns into theIllegalStateExceptionwhen there is no cached list. The binary download also catches it, so the partial archive is deleted and anIOExceptionis thrown.One
HttpClient/WebClientper process instead of one per call. They live in a lazy holder, so they are created on the first download and not when the class loads.spring-boot-starter-webfluxisprovided, so class loading behaves as before.Behaviour change:
proxyWithSystemProperties()is now read once, on the first download, instead of on every download.Tests
TerraformDownloaderTesthas three new tests that use an endpoint which never answers:hangingFetchWithoutAPreviousListFailsInsteadOfBlockingForever: expectsIllegalStateException.previousListIsUsedWhenARefreshHangs: expects the previous list to be returned.hangingResponseIsCutByTheResponseTimeout: only the response timeout is short, so this checks it is applied per request.The first two run with a 500ms releases timeout. With
.block()restored, both fail by timing out after 10s.mvn verify -Dgpg.skip: all modules pass, including the starter sample (35 tests interraform-client).The
OutOfDirectMemoryErrorpath itself is not reproduced in a unit test. It needs a constrained JVM; see the reproduction in the issue. The timeout is what unblocks that case, because the error never reachesblock().🤖 Generated with Claude Code