Skip to content

fix(downloader): resolve OpenTofu versions beyond the newest 30 releases - #120

Merged
alfespa17 merged 1 commit into
masterfrom
fix/tofu-release-pagination
Nov 18, 2026
Merged

alfespa17 merged 1 commit into
masterfrom
fix/tofu-release-pagination

Conversation

@Macbet

@Macbet Macbet commented Nov 18, 2026

Copy link
Copy Markdown

Stacked on #117 (fix/client-drop-webflux). Review only the last commit. Merge #117 first.

Problem

OpenTofu releases are read from https://api.github.com/repos/opentofu/opentofu/releases as a single request, with no per_page and no pagination. GitHub returns 30 items, so a version outside the newest 30 releases can't be resolved unless it is already installed. On 2026-11-18, resolveTofuVersion("1.8.0"), ("1.6.0") (the README example) and ("~>1.9.0") all threw Invalid tofu version range.

Change

  • Pagination. The downloader follows Link: rel="next" for every host, parsing rel as a token across all Link headers. Relative links are resolved against the final URL after redirects. It caps at 20 pages, stops on loops, and refuses an https to http downgrade.
  • per_page=100 only for GitHub. It is added only for api.github.com and GitHub Enterprise /api/v3/, and only when the URL doesn't set it already. Custom mirrors are requested exactly as configured, because pre-signed S3/GCS URLs sign the query string.
  • One total deadline. The existing releases download timeout covers all pages together, so the release-cache lock is held no longer than before.
  • Partial lists. If a later page fails (rate limit, I/O error, bad JSON, deadline), the pages already read are kept. They are merged with the previously cached list: fetched entries win, older versions are kept. The result is cached with the short retry TTL instead of 30 minutes. If page 1 fails, behaviour is unchanged.
  • Matching uses tag_name, falling back to name, with v prefix normalisation. Drafts are skipped. Prereleases behave as before.
  • Paths unchanged. Install directories stay tofu/v1.x.y/, so existing installs are reused.

Tests

  • New tests in TerraformDownloaderTest:
    • two pages on github.com and on /api/v3/;
    • per_page rules: mirror URL untouched, fragment, existing per_page;
    • page cap, loop, separate Link headers with a next-page decoy;
    • total deadline across slow pages;
    • 403 on page 2 keeps page 1;
    • a partial refresh merged with the cached list;
    • tag_name over name, drafts ignored;
    • an existing tofu/v1.x.y install reused.
  • 5 of the original 6 new tests fail on the previous code. The sixth guards the single-page mirror path, which should pass either way.
  • Docker verify (temurin 17, --init): 54/54 in terraform-client. The downloader tests were also repeated with --cpus=1.
  • Live GitHub: 1.6.0 resolves to v1.6.0, ~>1.8.0 to v1.8.11, ~>1.9.0 to v1.9.4.

@alfespa17

alfespa17 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

@Macbet this will require some changes in the executor proxy too, the library is used inside the executor component but it is not really calling the github endpoint there is one proxy that saves the responses there in redis.

https://github.com/terrakube-io/terrakube/blob/main/api/src/main/java/io/terrakube/api/plugin/json/TofuJsonController.java

Can you make sure to include support there too to manage different pages?

Base automatically changed from fix/client-drop-webflux to master November 18, 2026 22:45
The GitHub releases API returns 30 releases per page, and the tofu release
list was read with a single request, so any version older than the newest
30 releases (e.g. 1.6.0, ~>1.8.0) could not be resolved.

- Follow the Link rel="next" header (all Link headers, rel matched as a
  token, relative links resolved against the final URL after redirects)
  and merge all pages into the one cached list. A response without a next
  link, such as a static mirror, is used as is.
- Ask for per_page=100 only for GitHub (api.github.com or a GitHub
  Enterprise /api/v3/ path) and only when the URL has no per_page. Other
  URLs are requested exactly as configured, since pre-signed S3/GCS URLs
  sign their query string.
- Stop paging, with a warning, after 20 pages, on a link back to a page
  already read, or on a link from https to http.
- The releases download timeout caps all pages together, so paging never
  holds the releases lock longer than a single download could.
- If a later page fails (rate limit, I/O error, deadline), keep the pages
  read so far, merged with the previous cached list (de-duplicated by
  version, fetched entries win) so older releases are not lost, and cache
  it only as long as a failed fetch so it is refreshed soon. A failing
  first page behaves as before.
- Match on tag_name instead of the free-text release name, falling back
  to name for mirrors without tags, and compare versions without the v
  prefix. The resolved version is still the raw tag, so install
  directories (e.g. tofu/v1.13.9) stay the same.
- Ignore draft releases. Prereleases are kept as before; whether a range
  matches one is left to semver.
- Pick the first matching release instead of requiring exactly one, since
  a release published while paging can appear on two pages.
@alfespa17
alfespa17 force-pushed the fix/tofu-release-pagination branch from aae0fbc to 16f3672 Compare November 18, 2026 22:45
@sonarqubecloud

Copy link
Copy Markdown

@alfespa17
alfespa17 merged commit c992eb6 into master Nov 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants