Skip to content

Document policies for automated access/rate limits#13056

Merged
ericholscher merged 4 commits into
mainfrom
davidfischer/rate-limiting-page
Jun 1, 2026
Merged

Document policies for automated access/rate limits#13056
ericholscher merged 4 commits into
mainfrom
davidfischer/rate-limiting-page

Conversation

@davidfischer
Copy link
Copy Markdown
Contributor

@davidfischer davidfischer commented May 21, 2026 •
edited
Loading

  • Rate limits
  • Agent usage
  • Cross link to Markdown support
  • Caching
  • (Almost) no automated account creation

If merged and accepted, we can link to this from our default robots.txt in a comment.

Preview

https://readthedocs-landing--13056.org.readthedocs.build/platform/13056/automated-access.html

@davidfischer
Document policies for automated access
4ace512 
- Rate limits
- Agent usage
- Cross link to Markdown support
- Caching
- (Almost) no automated account creation
@davidfischer
Minor grammar fixes
68ac734
@davidfischer davidfischer requested a review from a team as a code owner May 21, 2026 00:55
@davidfischer davidfischer requested a review from ericholscher May 21, 2026 00:55
@read-the-docs-community
Copy link
Copy Markdown

read-the-docs-community Bot commented May 21, 2026 •
edited
Loading

Documentation build overview

📚 docs | 🛠️ Build #32795819 | 📁 Comparing f121740 against latest (e08ef8f)

  🔍 Preview build  

10 files changed · + 1 added · ± 9 modified

+ Added

± Modified

@davidfischer davidfischer changed the title Document policies for automated access Document policies for automated access/rate limits May 21, 2026
@davidfischer
Copy link
Copy Markdown
Contributor Author

davidfischer commented May 21, 2026

The vale linter doesn't like "ETag" which I've added here: readthedocs/common#303.

ericholscher
ericholscher reviewed May 21, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@ericholscher ericholscher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems mostly fine, but not super actionable? I'm not sure what I should be doing as a user after reading this page.

Comment thread docs/user/automated-access.rst Outdated
@davidfischer @ericholscher
Update docs/user/automated-access.rst
f121740 
Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com>
@davidfischer
Copy link
Copy Markdown
Contributor Author

davidfischer commented May 21, 2026

This seems mostly fine, but not super actionable? I'm not sure what I should be doing as a user after reading this page.

I think if we link it from the default robots.txt, it is actionable. I don't think anybody will find this page through any other means as it is pretty buried. The page states what our policies are and how the pages in question can be scraped. It is clear policies for users and agents to follow when scraping RTD. Is there another place these kind of policies should go?

@ericholscher
Copy link
Copy Markdown
Member

ericholscher commented May 22, 2026

I think it's fine -- I imagine most people won't read it so I don't want to spend too much time improving it with examples, but we should definitely link it from useful places.

@ericholscher ericholscher merged commit 28fb607 into main Jun 1, 2026
10 of 11 checks passed
@ericholscher ericholscher deleted the davidfischer/rate-limiting-page branch June 1, 2026 18:51
@ericholscher
Copy link
Copy Markdown
Member

ericholscher commented Jun 1, 2026

Merging to get it into stable this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericholscher ericholscher ericholscher approved these changes

Assignees

@davidfischer davidfischer

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@davidfischer @ericholscher