Skip to content

fix(download-ref): don't stall on Semantic Scholar 429s; hand chemistry PDFs to a person - #81

Merged
GiggleLiu merged 1 commit into
QuantumBFS:mainfrom
hz-xiaxz:fix/s2-ratelimit-fallback
Nov 4, 2026
Merged

GiggleLiu merged 1 commit into
QuantumBFS:mainfrom
hz-xiaxz:fix/s2-ratelimit-fallback

Conversation

@hz-xiaxz

@hz-xiaxz hz-xiaxz commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Problem

Adding JCP/AIP-style references often looked stuck. Without SEMANTIC_SCHOLAR_API_KEY, the shared S2 pool returns 429 for minutes at a time, and post_batch backed off 5+10+20+40+80 s per call. When it gave up:

  • arXiv IDs got no metadata at all (miss arxiv:…);
  • Crossref-only DOIs lost their externalIds.ArXiv, so the free-preprint fallback never ran.

In one real session, agents re-ran the fetch several times and adding about 10 papers took over 30 minutes.

On top of that, AIP (10.1063), ACS (10.1021) and RSC (10.1039) answer scripts and headless browsers with a Cloudflare "Just a moment..." page, even from a subscribed network. APS (10.1103) serves the PDF directly to subscribed IPs, and most APS papers have arXiv versions, which is why physics batches don't hit this.

Changes

  • Fail fast: post_batch gives up after about 9 s (two retries) and suggests setting SEMANTIC_SCHOLAR_API_KEY.
  • arXiv metadata fallback: arXiv IDs that S2 didn't return are looked up in batch on export.arxiv.org. Records carry title, authors, year, abstract, journal DOI and @misc BibTeX with eprint, and are marked metadata_source: "arxiv"; verify_bib._load_cache accepts these.
  • Preprint by title: Crossref-only DOIs search arXiv for an exact (normalized) title match and record externalIds.ArXiv.
  • Chemistry publishers: DOI misses under 10.1063/, 10.1021/ and 10.1039/ are printed as links, through SCIBRAIN_PROXY_PREFIX when set. The new helpers/import_pdfs.py scans recent PDFs (default ~/Downloads, last 14 days), matches each to a pending DOI by the DOI printed in it (with a boundary check) or by exact title, and copies it into .raw/doi/<safe>.pdf. Unmatched files are left alone.
  • Docs: SKILL.md and references/acquisition.md describe the flow.

Verification

  • Full suite passes: 277 tests. Five tests are new (fail-fast, arXiv fallback, preprint by title, chemistry-only link list, PDF import); the Crossref fallback test now expects the extra arXiv title query.
  • Live run on 2 arXiv IDs and 3 JCP DOIs with S2 forced to 429: finished in 39 s with the same result as an S2-available run. 10.1063/5.0157512 got its PDF via the preprint it found by title; the other two had no preprint and were listed as links.
  • import_pdfs.py was run against a real KB. The final step, a person downloading AIP PDFs, has not been exercised yet.

🤖 Generated with Claude Code

…ry PDFs to a person

Without an API key the shared Semantic Scholar pool often returns 429 for
minutes. post_batch backed off 5+10+20+40+80 s per call, and when it gave up,
arXiv IDs had no metadata at all and Crossref-only DOIs lost their preprint
link. Agents re-ran the fetch and sessions sat for half an hour.

- post_batch gives up after ~9 s (two retries) and points at
  SEMANTIC_SCHOLAR_API_KEY.
- arXiv IDs fall back to the arXiv API (title, authors, abstract, DOI,
  BibTeX; metadata_source "arxiv", accepted by verify_bib).
- Crossref-only DOIs look up an arXiv preprint by exact title.
- AIP (10.1063), ACS (10.1021) and RSC (10.1039) serve PDFs only after a
  Cloudflare human check, even to subscribed IPs. Those misses are now
  printed as links (through SCIBRAIN_PROXY_PREFIX when set), and the new
  import_pdfs.py matches the PDFs a person saves to the pending DOIs by
  DOI or exact title and copies them into .raw/doi/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GiggleLiu

Copy link
Copy Markdown
Member

LGTM, thank you!

@GiggleLiu
GiggleLiu merged commit a9abe78 into QuantumBFS:main Nov 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants