Skip to content

Add Lightning Model Registry fallback for gated HF tokenizer tests#2262

Draft
bhimrazy wants to merge 9 commits into
Lightning-AI:mainfrom
bhimrazy:ci/lightning-registry-tokenizer-fixtures
Draft

Add Lightning Model Registry fallback for gated HF tokenizer tests#2262
bhimrazy wants to merge 9 commits into
Lightning-AI:mainfrom
bhimrazy:ci/lightning-registry-tokenizer-fixtures

Conversation

@bhimrazy

@bhimrazy bhimrazy commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Fork PR CI currently breaks on one test. test_tokenizer_against_hf downloads tokenizer/config files for every supported model directly from Hugging Face.

About 40 of those repos are gated behind license acceptance (Llama, Gemma, Falcon-180B, Mistral-Large, ...). Since #2253 removed pull_request_target, fork PRs have no HF_TOKEN, so a single gated repo fails the whole CPU job.

This PR adds a fallback so the test resolves tokenizer/config files like this:

                Hugging Face Hub
              (uses HF_TOKEN if set)
                       |
        public repo, or gated + HF_TOKEN
                       |--------------------> use it
                       |
              gated + no token (fork PRs)
                       v
            Lightning Model Registry
           (public mirror, anonymous)
                       |
              fixture exists -----------------> use it
              no fixture ----------------------> skip test

tests/_fixtures.py implements this fallback and is used by test_tokenizer.py.

tests/publish_fixtures.py is a maintainer-only script that mirrors tokenizer/config files (never weights) for the gated repos to the lightning-ai/oss-litgpt registry teamspace.

A spot-check of one published fixture (meta-llama/Llama-2-7b-hf) confirms it resolves anonymously via the registry — no HF_TOKEN or Lightning login needed.

Internal PRs and main runs are unchanged — they keep using HF_TOKEN against Hugging Face directly.

No breaking changes.

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs) — follows from security: remove pull_request_target trigger from CPU tests workflow #2253, which flagged the need for a better solution than dropping fork secrets.
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary) — n/a, test-only change.
  • Did you write any new necessary tests? (not for typos and docs) — new resolver helper + reworked test_tokenizer.py.
  • Did you verify new and existing tests pass locally with your changes? — public and gated (fallback) paths verified locally; lint passes.
  • Did you list all the breaking changes introduced by this pull request? — none.
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors) — n/a, test-only.

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

…epos

Fork PRs run under `pull_request` and have no `HF_TOKEN`, so tokenizer parity
tests against gated-but-public HF repos (Llama, Gemma, etc.) fail there.

Add a CI-safe resolver that downloads tokenizer/config files from Hugging Face,
falls back to a public Lightning Model Registry mirror when a repo is gated and
no token is available (the lightning_sdk client uses an anonymous guest login),
and skips with a clear message when neither source works. Internal/main runs
keep full HF coverage via HF_TOKEN.

- tests/_fixtures.py: resolver + gated repo -> registry fixture map
- tests/test_tokenizer.py: use the resolver instead of direct HF downloads
- tests/upload_gated_tokenizer_fixtures.py: maintainer-only mirror upload script
- pyproject.toml: add litmodels to the test extra
@bhimrazy bhimrazy changed the title Add Lightning Model Registry fallback for gated HF tokenizer tests [wip]: Add Lightning Model Registry fallback for gated HF tokenizer tests Jun 11, 2026
@bhimrazy bhimrazy marked this pull request as draft June 11, 2026 07:34
bhimrazy added 2 commits June 16, 2026 00:26
Use the verified `lightning-ai/oss-litgpt` teamspace for tokenizer/config
fixtures, and rewrite the maintainer-only mirroring script as
tests/publish_fixtures.py, parallelizing repo uploads with litdata.map.
@bhimrazy bhimrazy force-pushed the ci/lightning-registry-tokenizer-fixtures branch from 9d3ba91 to 6c9295d Compare June 15, 2026 18:52
@bhimrazy bhimrazy changed the title [wip]: Add Lightning Model Registry fallback for gated HF tokenizer tests Add Lightning Model Registry fallback for gated HF tokenizer tests Jun 15, 2026
bhimrazy and others added 5 commits June 16, 2026 00:45
Adds [fixtures] markers for HF vs registry-fallback resolution and runs the
scoped tokenizer job with -rs/-s so CI shows which path each repo took and why
any skip happens. Temporary, paired with the test_tokenizer.py scoping.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant