Repository navigation
fix(downloader): resolve OpenTofu versions beyond the newest 30 releases - #120
Merged
Merged
Conversation
Member
|
@Macbet this will require some changes in the executor proxy too, the library is used inside the executor component but it is not really calling the github endpoint there is one proxy that saves the responses there in redis. Can you make sure to include support there too to manage different pages? |
The GitHub releases API returns 30 releases per page, and the tofu release list was read with a single request, so any version older than the newest 30 releases (e.g. 1.6.0, ~>1.8.0) could not be resolved. - Follow the Link rel="next" header (all Link headers, rel matched as a token, relative links resolved against the final URL after redirects) and merge all pages into the one cached list. A response without a next link, such as a static mirror, is used as is. - Ask for per_page=100 only for GitHub (api.github.com or a GitHub Enterprise /api/v3/ path) and only when the URL has no per_page. Other URLs are requested exactly as configured, since pre-signed S3/GCS URLs sign their query string. - Stop paging, with a warning, after 20 pages, on a link back to a page already read, or on a link from https to http. - The releases download timeout caps all pages together, so paging never holds the releases lock longer than a single download could. - If a later page fails (rate limit, I/O error, deadline), keep the pages read so far, merged with the previous cached list (de-duplicated by version, fetched entries win) so older releases are not lost, and cache it only as long as a failed fetch so it is refreshed soon. A failing first page behaves as before. - Match on tag_name instead of the free-text release name, falling back to name for mirrors without tags, and compare versions without the v prefix. The resolved version is still the raw tag, so install directories (e.g. tofu/v1.13.9) stay the same. - Ignore draft releases. Prereleases are kept as before; whether a range matches one is left to semver. - Pick the first matching release instead of requiring exactly one, since a release published while paging can appear on two pages.
alfespa17
force-pushed
the
fix/tofu-release-pagination
branch
from
November 18, 2026 22:45
aae0fbc to
16f3672
Compare
|
alfespa17
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
OpenTofu releases are read from
https://api.github.com/repos/opentofu/opentofu/releasesas a single request, with noper_pageand no pagination. GitHub returns 30 items, so a version outside the newest 30 releases can't be resolved unless it is already installed. On 2026-11-18,resolveTofuVersion("1.8.0"),("1.6.0")(the README example) and("~>1.9.0")all threwInvalid tofu version range.Change
Link: rel="next"for every host, parsingrelas a token across allLinkheaders. Relative links are resolved against the final URL after redirects. It caps at 20 pages, stops on loops, and refuses an https to http downgrade.per_page=100only for GitHub. It is added only forapi.github.comand GitHub Enterprise/api/v3/, and only when the URL doesn't set it already. Custom mirrors are requested exactly as configured, because pre-signed S3/GCS URLs sign the query string.tag_name, falling back toname, withvprefix normalisation. Drafts are skipped. Prereleases behave as before.tofu/v1.x.y/, so existing installs are reused.Tests
TerraformDownloaderTest:/api/v3/;per_pagerules: mirror URL untouched, fragment, existingper_page;Linkheaders with anext-pagedecoy;tag_nameovername, drafts ignored;tofu/v1.x.yinstall reused.verify(temurin 17,--init): 54/54 in terraform-client. The downloader tests were also repeated with--cpus=1.1.6.0resolves tov1.6.0,~>1.8.0tov1.8.11,~>1.9.0tov1.9.4.